Llmstxt Doc Search

llmstxt-doc-search

Live, ranked search across any number of llms.txt documentation sites - Strands, Kiro, the AWS guides, and whatever you add at runtime.

llmstxt-doc-search is a Model Context Protocol (MCP) server that turns the llms.txt index a documentation site publishes into a fast, ranked search tool your agent can call. It indexes titles at startup, ranks queries with BM25, and fetches the full document only when you open a result - so you get current docs with almost no local storage. Built on the search engine from @praveenc/mcp-docs-server, generalized to a runtime registry of sources.

It implements the MCP 2026-07-28 specification over stdio and still works with clients on the 2025 protocol, which open with an initialize handshake. Requires Node.js 20 or later.

---

Why

An llms.txt file is a curated index of a doc site's pages, published for tools like this one to consume. They can be large - AWS Bedrock's lists roughly a thousand documents - so downloading everything is wasteful and goes stale fast.

This server takes a leaner approach:

• Title-only index, built lazily. On first search of a source, only the page titles are indexed. That is fast to build and tiny to hold in memory.
• Ranked with BM25. Queries are scored with BM25 plus Porter stemming, bigrams, and markdown-aware weighting (headers, code, and links count for more). Technical terms like mcp, json, and stdio are preserved rather than stemmed.
• Content on demand. The full markdown or HTML of a result is fetched only when you call fetch_doc.

The result is a good fit for broad, fast-moving reference material - the opposite tradeoff to snapshotting docs into a local vault.

---

Installation

Quick start (recommended)

Add the server to your MCP client configuration (Claude Desktop, Kiro, and others). It is downloaded and run on demand via npx - no manual build:

{

"mcpServers": { "llmstxt-doc-search": { "command": "npx", "args": ["-y", "@praveenc/llmstxt-doc-search"] } } }

Global install

npm install -g @praveenc/llmstxt-doc-search

Then point your MCP client at the installed binary:

{

"mcpServers": { "llmstxt-doc-search": { "command": "llmstxt-doc-search" } } }

---

Quick start

Once the server is connected, the typical flow is three calls:

1. `docs_home()` - orient yourself: see the registered sources and how to search and fetch.
2. `search_docs("prompt caching", "aws-bedrock-userguide")` - rank matching docs. Omit the source to search everything.
3. `fetch_doc(url)` - read the full content of a result you like.

Add your own source at any time and it is indexed immediately and persisted for future runs:

add_doc_source("langgraph", "https://langchain-ai.github.io/langgraph/llms.txt")

---

Tools

docs_home, list_doc_sources, search_docs, and fetch_doc are annotated read-only, so a client can approve them without prompting. add_doc_source, remove_doc_source, and refresh_doc_source change the persisted registry, and remove_doc_source is annotated destructive. The tool list is fixed, so it is advertised as cacheable for one hour.

Default sources

Seeded into the registry on first run:

strands, kiro, aws-bedrock-userguide, aws-agentic-ai-lens, aws-bedrock-agentcore-devguide, mcp.

The registry is persisted at ~/.config/llmstxt-doc-search/sources.json (override with LLMSTXT_REGISTRY_PATH). Anything you add, remove, or refresh at runtime is saved there.

---

Configuration

All configuration is via environment variables; none are required.

---

Testing with MCP Inspector

npx @modelcontextprotocol/inspector npx -y @praveenc/llmstxt-doc-search

The Inspector can connect in either protocol era; see Protocol eras.

---

Development

Clone the repository for local work (Node.js 20 or later):

git clone https://github.com/praveenc/llmstxt-doc-search.git

cd llmstxt-doc-search npm install

Commands

npm run dev         # run from source with tsx (no build)

npm test # offline unit and protocol tests npm run typecheck # type-check without emitting npm run build # compile to dist/ npm run inspect:dev # MCP Inspector against the source

Local MCP client config (development)

Point your client at a source checkout instead of the published package:

{

"mcpServers": { "llmstxt-doc-search": { "command": "npx", "args": ["tsx", "/ABS/PATH/llmstxt-doc-search/src/index.ts"] } } }

Or, after npm run build, at the compiled entry point:

{

"mcpServers": { "llmstxt-doc-search": { "command": "node", "args": ["/ABS/PATH/llmstxt-doc-search/dist/index.js"] } } }

---

Architecture

src/

├── index.ts # Tool registration and the stdio entry point (serves 2026-07-28 and 2025-era clients) ├── config.ts # Defaults and environment configuration ├── tools/ │ └── docs.ts # search_docs, fetch_doc, and source management └── utils/ ├── doc-fetcher.ts # HTTP fetching, redirect handling, HTML parsing ├── indexer.ts # BM25 search index ├── registry.ts # Persisted source registry ├── store.ts # In-memory document store ├── text-processor.ts # Tokenization and snippet helpers ├── url-validator.ts # SSRF guard and URL validation ├── stopwords.ts # Stop-word list └── logger.ts # Logging utilities

---

Search algorithm

Ranking uses BM25 (Best Matching 25) with several enhancements:

• Porter stemming matches word variants (for example, running and run).
• Bigrams capture phrase matches (for example, prompt caching).
• Weighted scoring boosts title matches (3-8x), headers (4x), code blocks (2x), and link text (2x).
• Domain-term preservation keeps technical terms like mcp, json, and stdio unstemmed so they match exactly.

---

Security

This server fetches user-supplied URLs at runtime, so its SSRF surface is guarded in depth:

• Scoped fetches. fetch_doc only retrieves URLs that a registered source's llms.txt lists exactly, or that sit under that source's origin and path prefix (matched on a path boundary rather than a raw string prefix). There is no arbitrary fetch.
• Scheme allow-list. Non-http(s) schemes are rejected.
• Range-based address blocking. Private and reserved destinations are blocked using IP range classification (ipaddr.js), covering decimal, octal, and hex IPv4, IPv4-mapped IPv6, loopback, link-local, unique-local, carrier-grade NAT, and other reserved ranges - not just a hostname regex.
• Connection-time validation. The resolved IP is checked at connection time via a custom DNS lookup, closing DNS-rebinding, and every redirect hop is re-validated.
• Bounded responses. Response bodies are capped at 10 MB to limit memory and regular-expression (ReDoS) exposure.

Runtime dependencies report zero known vulnerabilities.

---

License

MIT - Copyright (c) 2026 Praveen Chamarthi

---

Contributing

Contributions are welcome. If you find a bug or have an idea:

1. Open an issue describing the problem or proposal.
2. For code changes, fork the repo and create a feature branch.
3. Keep changes focused, add or update tests, and make sure npm test, npm run typecheck, and npm run build all pass.
4. Open a pull request against main with a clear description of what changed and why.

Commit messages follow the Conventional Commits style.

---

Support

• Questions and ideas: open a GitHub issue.
• Bugs: please include your MCP client, the tool call you made, and any relevant logs (set LLMSTXT_LOG_LEVEL=debug for more detail).
• Security issues: open an issue marked as security-sensitive, or contact the maintainer directly rather than posting exploit details publicly.

---

Built for the MCP community ❤️