Documentation Extractor

llm.energy

Extract llms.txt documentation and install.md instructions from any website for AI agents, LLMs, and automation workflows.

llm.energy

📖 Overview

llm.energy is a web application and MCP server that fetches, parses, and organizes documentation from websites implementing the llms.txt and install.md standards. It transforms raw documentation into structured, agent-ready formats optimized for large language models, AI assistants, and developer tooling.

---

✨ Key Features

---

🎯 Use Cases

─────────────────────────────────────────────────────────────────

📝 Feed docs to AI coding assistants (Cursor, Windsurf) 🤖 Build context-aware AI agents with up-to-date docs 🔄 Create documentation pipelines for RAG systems 📦 Aggregate docs from multiple sources automatically ✏️ Generate llms.txt/install.md for your own projects 🚀 Auto-generate install.md from any GitHub repo

─────────────────────────────────────────────────────────────────

---

🚀 install.md Generator

Generate LLM-executable installation instructions from any source:

How It Works

1. GitHub Mode: Analyzes README, package.json/pyproject.toml/Cargo.toml, GitHub Actions, and releases
2. URL Mode: Scrapes documentation pages, detects platforms (Mintlify, Docusaurus, GitBook, etc.)
3. AI Synthesis: Uses Claude to generate a properly formatted install.md

Quick Example

# Generate install.md for any GitHub project

curl -X POST https://llm.energy/api/generate-install \ -H "Content-Type: application/json" \ -d '{"url": "https://github.com/anthropics/anthropic-sdk-python", "type": "github"}'

Visit [llm.energy/install-generator](https://llm.energy/install-generator) to use the web interface.

---

📚 Documentation

Full documentation is available at [llm.energy/docs](https://docs.llm.energy/)

---

🚀 Installation

Web Application

Visit llm.energy to use the hosted version.

Local Development

# Clone the repository

git clone https://github.com/nirholas/extract-llms-docs.git cd extract-llms-docs

Install dependencies

pnpm install

Start development server

pnpm dev

🌐 The application runs on http://localhost:3001

MCP Server

📦 Click to expand MCP Server setup

Add to your MCP client configuration (Claude Desktop, Cursor, etc.):

{

"mcpServers": { "llm-energy": { "command": "npx", "args": ["-y", "@llm-energy/mcp-server"] } } }

See MCP Server Documentation for detailed setup.

---

🔌 API Reference

POST /api/extract - Extract documentation from a URL

curl -X POST https://llm.energy/api/extract \

-H "Content-Type: application/json" \ -d '{"url": "docs.anthropic.com"}'

Response includes parsed sections, metadata, and download URLs.

POST /api/validate - Check if URL has llms.txt support

curl -X POST https://llm.energy/api/validate \

-H "Content-Type: application/json" \ -d '{"url": "docs.example.com"}'

POST /api/batch - Process multiple URLs

curl -X POST https://llm.energy/api/batch \

-H "Content-Type: application/json" \ -d '{"urls": ["docs.anthropic.com", "docs.stripe.com"]}'

GET /api/download - Download in various formats

# Formats: markdown, json, yaml, zip

curl "https://llm.energy/api/download?url=docs.anthropic.com&format=zip"

POST /api/generate-install - AI-generate install.md from GitHub or docs URL

# Generate from a GitHub repository

curl -X POST https://llm.energy/api/generate-install \ -H "Content-Type: application/json" \ -d '{"url": "https://github.com/anthropics/anthropic-sdk-python", "type": "github"}'

Generate from a documentation URL

curl -X POST https://llm.energy/api/generate-install \ -H "Content-Type: application/json" \ -d '{"url": "https://docs.anthropic.com/en/docs/quickstart", "type": "docs"}'

Analyze only (preview mode)

curl "https://llm.energy/api/generate-install?url=https://github.com/anthropics/anthropic-sdk-python&type=github"

📖 [Full API Reference →](https://docs.llm.energy/api-reference/)

---

🤖 MCP Server Tools

The MCP server exposes the following tools for AI agents:

---

🏗️ Architecture

┌──────────────────────────────────────────────────────────────────┐

│ llm.energy │ ├──────────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Web App │ │ REST API │ │ MCP Server │ │ │ │ (Next.js) │ │ /api/* │ │ (stdio) │ │ │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ │ │ │ │ └────────────────┼────────────────┘ │ │ │ │ │ ┌──────▼──────┐ │ │ │ Core │ │ │ │ - Parser │ │ │ │ - Extractor│ │ │ │ - Cache │ │ │ └──────┬──────┘ │ │ │ │ │ ┌────────────────┼────────────────┐ │ │ ▼ ▼ ▼ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ llms.txt │ │ install.md │ │ Sitemap │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │ └──────────────────────────────────────────────────────────────────┘

📁 Project Structure

extract-llms-docs/

├── src/ │ ├── app/ # Next.js pages and API routes │ │ ├── api/ # REST API endpoints │ │ │ ├── extract/ # Documentation extraction API │ │ │ ├── generate-install/ # AI-powered install.md generation │ │ │ └── ... # Other API endpoints │ │ ├── extract/ # Extraction interface │ │ ├── batch/ # Batch processing page │ │ ├── directory/ # Site directory browser │ │ ├── generator/ # llms.txt generator wizard │ │ └── install-generator/ # install.md generator with tabs │ ├── components/ # React UI components │ │ └── install-generator/ # GitHubTab, UrlTab, Preview │ ├── lib/ # Core utilities │ │ ├── github-analyzer.ts # GitHub repo analysis │ │ ├── docs-analyzer.ts # Documentation URL scraping │ │ └── ... # Parser, extractor, cache │ ├── hooks/ # React hooks │ └── types/ # TypeScript definitions ├── packages/core/ # Shared parser and types ├── mcp-server/ # MCP server package └── docs-site/ # MkDocs documentation source

---

🛠️ Technology Stack

---

⚙️ Configuration

All environment variables are optional — the app runs without them. Copy .env.example to .env.local and fill in only what you need:

cp .env.example .env.local

| ADMIN_KEY | /api/cache | _unset_ | Value the x-admin-key header must match for cache management. Both GET and POST return 500 when it is