Webshift

WebShift

---

What is WebShift

WebShift is a Rust library and MCP server that shifts noisy web pages into clean, right-sized text for LLM consumption.

Raw HTML is mostly junk: scripts, ads, navigation menus, cookie banners, tracking pixels. Feeding it directly to an LLM floods the context window with tens of thousands of useless tokens and leaves no room for reasoning. WebShift strips all that noise, sterilizes the text, and enforces strict size budgets so the model receives only the content that matters.

What you get

Depending on the features you enable, WebShift can be four things:

When to use WebShift

• You're building an AI agent that needs web search and you want clean, budget-controlled text — not raw HTML.
• You're processing web pages in a Rust pipeline and need a reliable HTML-to-text cleaner that strips noise without losing real content.
• You need an LLM to translate, rewrite, or simplify text inside HTML without corrupting the markup — text-map gives you a safe round-trip.
• You want an MCP web search server that works as a single binary — no Python, no pip, no venv, no Docker (unless you want it).
• You need hard guarantees on output size: per-page caps, total budget caps, streaming download limits.

When NOT to use WebShift

• You need a headless browser that renders JavaScript-heavy SPAs. WebShift parses static HTML — it doesn't execute JS.
• You need to render or screenshot a page preserving its visual layout. WebShift processes HTML structure but does not render CSS or compute layout. (Note: text-map does preserve the DOM structure for text rewriting use cases.)
• You're building a web scraper that needs to navigate across pages, fill forms, or handle authentication flows.

---

How it works

Question

| +- (optional) LLM query expansion -> multiple search variants | +- Search via backend (SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing, HTTP) | +- Deduplicate + filter binary URLs | +- Streaming fetch with per-page size cap | +- HTML cleaning -> plain text (noise elements, scripts, nav removed) | +- Unicode/BiDi sterilization | +- BM25 deterministic reranking | +- (optional) LLM-assisted tier-2 reranking | +- Budget-aware truncation across all sources | +- (optional) LLM Markdown summary with inline citations | +- Structured JSON output

For a detailed explanation of each pipeline stage, BM25 parameters, adaptive budget allocation, and real compression metrics see Under the Hood. For the full configuration reference (TOML, env vars, CLI args) see Configuration. For ready-to-use examples see Use Cases.

---

Installation

Binary (MCP server)

cargo install webshift-mcp

The binary is called mcp-webshift.

From source

cargo install --path crates/webshift-mcp

As a library

# Full pipeline (search + fetch + clean + rerank)

webshift = "0.2"

Cleaner + fetcher only (no search backends)

webshift = { version = "0.2", default-features = false }

Text-map only (extract/replace text nodes in HTML)

webshift = { version = "0.2", default-features = false, features = ["text-map"] }

Everything including LLM features

webshift = { version = "0.2", features = ["llm"] }

---

Quick start

1. Set up a search backend

The easiest option is SearXNG — free, self-hosted, no API key:

docker run -d -p 8080:8080 searxng/searxng

No Docker? Use a cloud backend — see Search backends.

2. Configure your MCP client

{

"mcpServers": { "webshift": { "command": "mcp-webshift", "args": ["--default-backend", "searxng"] } } }

That's it. The agent now has webshift_query, webshift_fetch, and webshift_onboarding.

For client-specific setup see docs/integrations/.

---

MCP tools

webshift_query parameters

---

Configuration

Resolution order (highest priority first):

1. CLI args — --default-backend, --brave-api-key, etc.
2. Environment variables — WEBSHIFT_* prefix
3. Config file — webshift.toml (current dir, then ~/webshift.toml)
4. Built-in defaults

Config file

[server]

max_query_budget = 32000 # total char budget across all sources max_result_length = 8000 # per-page char cap max_total_results = 20 # hard cap on results per call max_download_mb = 1 # streaming cap per page (MB) search_timeout = 8 # seconds results_per_query = 5 oversampling_factor = 2 adaptive_budget = "auto" # "auto" | "on" | "off" — budget allocation mode

[backends] default = "searxng"

[backends.searxng] url = "http://localhost:8080"

[backends.brave] api_key = "BSA-..."

[backends.tavily] api_key = "tvly-..."

[backends.exa] api_key = "..."

[backends.serpapi] api_key = "..." engine = "google" # google | bing | duckduckgo | yandex

[backends.google] api_key = "..." cx = "..." # Custom Search Engine ID

[backends.bing] api_key = "..." market = "en-US"

[backends.http] url = "https://my-search.example.com/api/search" query_param = "q" count_param = "limit" results_path = "data.items" # dot-path to results array in JSON response title_field = "title" url_field = "link" snippet_field = "description"

[backends.http.headers] "Authorization" = "Bearer my-token"

[llm] enabled = false base_url = "http://localhost:11434/v1" # OpenAI-compatible api_key = "" model = "gemma3:27b" timeout = 60 expansion_enabled = true summarization_enabled = true llm_rerank_enabled = false

For every setting with all three config methods (TOML, env vars, CLI args) and plain-language descriptions, see the full Configuration Reference. Ready-to-use config examples are in Use Cases and examples/.

Key environment variables

WEBSHIFT_DEFAULT_BACKEND=searxng

WEBSHIFT_SEARXNG_URL=http://localhost:8080 WEBSHIFT_BRAVE_API_KEY=BSA-xxx WEBSHIFT_GOOGLE_API_KEY=xxx WEBSHIFT_GOOGLE_CX=xxx WEBSHIFT_BING_API_KEY=xxx WEBSHIFT_LLM_ENABLED=true WEBSHIFT_LLM_BASE_URL=http://localhost:11434/v1 WEBSHIFT_LLM_MODEL=gemma3:27b

---

Search backends

---

LLM features (optional)

All opt-in — disabled by default, no data leaves your machine unless enabled.

Cross-language normalization (bonus): when BM25 reranking surfaces pages in

foreign languages (e.g. Chinese, Japanese, Arabic), the LLM summarizer still

produces the final report in the prompt language. The agent receives clean,

readable output regardless of the language mix in the source pages.

Works with any OpenAI-compatible API (OpenAI, Ollama, vLLM, LM Studio, etc.):

[llm]

enabled = true base_url = "http://localhost:11434/v1" model = "gemma3:27b"

---

Anti-flooding protections

Always active — the core value proposition:

---

Library usage

use webshift::{Config, clean, fetch, query};

// Clean raw HTML — cap output at 8000