Token Context MCP

Token Context MCP (v0.3.6)

🌐 English: README.md | 🇻🇳 Tiếng Việt: README.vi.md | 🇨🇳 简体中文: README.cn.md | 📝 Prompt Protocol: sample_prompt.md | 📋 Changelog: CHANGELOG.md

[!NOTE]

Architecture Notice: The governance microkernel design is inspired by the Cordis paper (*A Programming Paradigm for Spatiotemporal Composability*, arXiv:2608.25512). This is an independent open-source implementation not affiliated with or endorsed by DeepSeek.

token-context-mcp is an on-demand, budget-governed Model Context Protocol (MCP) server for autonomous AI coding agents (Claude Code, Cursor, Codex, Google Antigravity). Instead of blindly dumping an entire repository or compressing it into a static snapshot that strips function bodies down to signatures, it provides an interactive AST-level pull pipeline that delivers exact code symbols and downstream impact graphs with verified file:line + SHA-256 evidence.

---

🖼️ Architecture & Benchmark Infographics (2x Retina Dark Green)

🔍 View detailed technical architecture & comparison diagrams (Click to expand)

1. 3-Layer Governance Microkernel & 5-Step Progressive Pull Pipeline

2. Context Paradigms: Flat Dump vs. Static Pack vs. Interactive MCP Pull

3. Empirical Benchmarks: 59-Task Head-to-Head & M10 Equal-Budget Suite

4. 7-Dimension Technical Comparison Matrix (vs. ContextCraft v0.3.2)

---

⚡ Core Architectural Pillars

1. 3-Stage Governance Microkernel (`kernel/`):

• Every request passes through a lightweight middleware pipeline:
• Stage 1 — Identity Resolution (`AuthMiddleware`): Binds TOKEN_CONTEXT_AGENT_ID when configured and prevents reserved admin impersonation.
• Stage 2 — Access & Rate Governance (`GovernanceMiddleware`): Enforces per-agent policies, emergency halt, and sliding-window rate limits.
• Stage 3 — AST-Boundary Cap & Execution (`ExecutionMiddleware` + `retrieve/`): Enforces max_result_tokens at syntactic boundaries without cutting functions in half.

1. Zero Schema Bloat:

• The default tools/list payload exposes only 10 core read-only retrieval tools (keeping 11 extension tools—including trajectory_record—and 2 admin tools off the wire until requested). Server-side JSON schemas are slimmed to reduce initial prompt tax.

1. Incremental Tree-sitter AST & Persistent SQLite Graph (`index/`):

• Fast mtime_ns + SHA-256 change detection across Python, JavaScript/TypeScript, Go, Rust, Java, C/C++, and C# with symlink-escape and path traversal guards.

1. Verified Evidence Ledger ("Lá Phổi" — `[ledger].enabled = true` by default):

• Stores verified "query -> answer" citation cards (path:line + AST body SHA-256) in ledger.sqlite outside your repository. Every read re-verifies citations against disk, emits prior_evidence hints (or recheck_from on VERIFY_FAILED), and never prunes unverified code. Zero-hash fast-path adds 0ms overhead on empty ledgers.

---

📊 Empirical Benchmarks (tiktoken o200k_base Audited)

All numbers are measured using OpenAI's o200k_base tokenizer across two real-world open-source repositories: rich (Python, 213 files, 507k tokens) and fastify (JS/TS, 291 files, 538k tokens):

| M10 Equal-Budget Suite (rich, n=30, ~1.9k tokens) | File Recall & Symbol Recall (95% Bootstrap CI) | 0.90 File Recall 0.54 Symbol Recall | 0.57 File Recall 0.10 Symbol Recall *(Truncated Grep)* | +58% file recall, 5.4× symbol recall over truncated grep at identical token cost | | C1 Single-Call Cost (fastify 538k source baseline) | Orientation map get_repo_map@512 | 682 tokens | 538,799 tokens (raw source files) | Compact repo map occupies 0.13% of the repository (~790× smaller) |

---

🚀 Quick Start

Option 1: Run instantly with uvx (Recommended)

# Register, index, and print agent config for the current directory

uvx token-context-mcp quickstart .

Option 2: Install via pip

pip install token-context-mcp

Register a codebase and launch server

token-context register --repo-id my-project --root /path/to/project token-context index --repo-id my-project token-context serve

Desktop GUI Controller (PySide6)

7-tab dark desktop controller (Dashboard, Repositories, Tasks & Graph, Cache & DB with stale snapshot cleanup & ledger.sqlite vacuuming, Ledger & Memory, Agents & Security, and Settings):

uvx --from "token-context-mcp[gui]" token-context-gui

Benchmark & Evaluation Suite ([eval] extra with tiktoken o200k_base)

pip install -e ".[eval]"

python evals/chain_eval.py --output evals/reports/chain_l0_l5.json python evals/mcp_chain_eval.py --limit-chains 6 --output evals/reports/mcp_chain_eval.json

---

🤖 Coding Agent Setup

token-context-mcp communicates over local stdio. Add the configuration below to your agent:

1. Claude Code

claude mcp add --transport stdio --scope user token-context -- uvx token-context-mcp serve

2. Cursor

In Cursor Settings > Features > MCP Servers:

• Name: token-context
• Type: command
• Command: uvx token-context-mcp serve

3. Claude Desktop

Add to %APPDATA%\Claude\claude_desktop_config.json (Windows) or ~/.config/Claude/claude_desktop_config.json (macOS/Linux):

{

"mcpServers": { "token-context": { "command": "uvx", "args": ["token-context-mcp", "serve"] } } }

4. Google Antigravity IDE / Gemini CLI

Add to mcp_config.json:

{

"mcpServers": { "token-context": { "command": "uvx", "args": [ "token-context-mcp", "serve", "--output-mode", "text", "--schema-profile", "gemini_safe" ] } } }

---

🛠 Tool Reference Catalog (10 Default Core Tools)

By default, the server exposes 10 read-only retrieval tools to preserve agent context:

*(11 extension tools including SQLite multi-agent memory memory_*, Evidence Ledger trajectory_record, lazy tool discovery *_tools, hardware-adaptive compressor sample_summarize, and 2 admin tools agent_control, audit_logs can be enabled via configuration).*

---

🔒 Security Guarantees & Privacy

• No Remote Code Execution (RCE): No tool can execute shell commands, eval code, or modify repository source files.
• 100% Local & Private: All SQLite WAL indexes remain on your machine; zero cloud telemetry or external API calls.
• Automatic Secret Redaction: Regex-based redaction of API keys, tokens, and private keys ([REDACTED: potential secret]) plus hard denial of sensitive files (.env*, .ssh, *.pem, id_rsa).

---

📚 Technical Documentation

• sample_prompt.md: Standard two-phase prompt templates (EN / VI / CN).
• CHANGELOG.md: Release history and upgrade notes.
• launch/00_fact_sheet.md: Verified empirical benchmarks and measurement methodologies.
• docs/CLIENT_MATRIX.md: Step-by-step MCP client verification matrix.

---

*Maintained by the Token Context MCP community. Licensed under MIT.*