MarkIndex

MarkIndex

Vectorless, structure-first retrieval over large libraries of long Markdown documents — for agents.

MarkIndex parses every Markdown file into a tree of its own sections, stores the trees, the original lines and a mixed Chinese/English BM25 index in one SQLite file, and gives an agent five read-only tools to walk "library → folder → document → section" and read the original text with line numbers. No embeddings, no vector database, no chunking guesswork: the document's own structure is the index, and every piece of evidence can be cited back as [doc_path, L - ].

Inspired by VectifyAI/PageIndex, MarkIndex is an independent,

unofficial TypeScript implementation focused on Markdown. It is not affiliated with or endorsed by VectifyAI.

中文说明：README.zh-CN.md

Why

• Structure-first, not chunk-first. Long professional documents (filings, guidelines, laws, manuals) are already organized by their authors. Navigating that outline keeps context intact and makes answers explainable and citable; arbitrary chunks + vector similarity cut sections apart.
• Deterministic and LLM-free indexing. Parsing (mdast, not regex), tree building and the token-cost tree optimization never call a model; the same input always produces the same tree. LLMs are optional and only used for node summaries, document descriptions and the built-in retrieve() loop.
• Markdown done properly. Setext headings, preface text before the first heading, documents without any heading, front matter, # inside code fences, skipped heading levels, BOM / CRLF — every source line belongs to exactly one node, and oversized sections are split at block boundaries into chunk nodes.
• Keyword fallback that works for Chinese. BM25 over Intl.Segmenter word segmentation (Chinese, English and mixed text, identifiers like C++, ISO-27001, v2.1), plus a trigram substring index as a second pass.
• Budgeted tool responses. Tools never throw; they return JSON envelopes with next_steps guidance and stay under a 95 000-character budget, paginating large outlines and sections.

Install

npm install markindex          # library + CLI (Node.js >= 22.12)

npx markindex --help

From source:

git clone https://github.com/jackjin1997/markindex.git && cd markindex

npm install && npm run build node dist/cli.js --help # or npm link to get a markindex command

Optional, only for LLM features (summaries, retrieve(), agent-mode evaluation): install the AI SDK provider you want, e.g. npm install @ai-sdk/anthropic, and set its API key (ANTHROPIC_API_KEY, …). With npx and no local install, put the provider in the same temporary environment:

npx -p markindex -p @ai-sdk/anthropic markindex index ./docs --summary-model anthropic:claude-sonnet-5-5

CLI

# Build or incrementally update the index (stored in <root>/.markindex/index.db)

markindex index ./docs markindex index ./docs --summary-model anthropic:claude-sonnet-5-5 --estimate # cost estimate, no LLM calls markindex index ./docs --summary-model anthropic:claude-sonnet-5-5 # with node summaries markindex index ./docs --summary-model anthropic:claude-sonnet-5-5 --summary-rpm 50 --summary-tpm 40000 # rate-limited markindex index ./docs --json # machine-readable report markindex index ./docs --no-setext # converter output with --- page breaks markindex index ./docs --ignore 'drafts/' --ignore '*.tmp.md' # extra ignore patterns (repeatable)

Inspect one document's section tree

markindex tree ./docs guides/setup.md --depth 3

Keyword search (BM25 + trigram fallback)

markindex search ./docs "离职 补偿" markindex search ./docs "ISO-27001 audit" --folder policies --limit 5 --json

Serve the library to agents over MCP (stdio, read-only)

markindex mcp ./docs markindex mcp ./docs --reindex-on-start

Measure retrieval quality on your own documents (see "Evaluation")

markindex eval ./docs --questions questions.jsonl --mode bm25 --out report.json markindex eval ./docs --questions questions.jsonl --mode agent --model anthropic:claude-sonnet-5-5 --out report.json markindex eval ./docs --questions questions.jsonl --mode agent --model anthropic:claude-sonnet-5-5 --rpm 50 --tpm 40000

Rate limits. Calls are always bounded by concurrency (--concurrency for eval; 8 for summaries) and the AI SDK's retries, which already honor retry-after / retry-after-ms on 429 / 5xx responses (up to 60 s). To stay under a provider quota, add a client-side limit: --summary-rpm / --summary-tpm on index, --rpm / --tpm on eval (agent mode). rpm caps requests and tpm caps *estimated input* tokens in any sliding 60 s window; both are shared by all documents (index) or all questions (eval) of the run, and waiting callers are served in order. Input tokens are estimated from the prompt text (about 1 token per CJK character, otherwise 4 characters), so leave headroom below the real quota; output tokens and tool definitions are not counted. A single request larger than the tpm still goes through, alone. In the library, pass rateLimit: { requestsPerMinute, inputTokensPerMinute } to summarizeDocument / indexLibrary({ summary }) or retrieve(); reuse the same object to share one budget.

--summary-model / --model take : ; MarkIndex imports @ai-sdk/ dynamically, so only the provider you use needs to be installed. Every command accepts --db to use a different index file. Put the index on a local disk: the writer lock uses hard links and SQLite runs in WAL mode, neither of which works reliably on exFAT, SMB / NFS shares or other network volumes. If the documents live there, keep them in place and point --db at a local path (e.g. --db ~/.cache/markindex/docs.db).

Indexing is incremental: files are compared by content hash, only new or changed documents are rebuilt (one transaction per document), deleted files are removed, and summaries are cached by node content hash. Scanned files are .md and .markdown (not .mdx). Ignored are: paths matched by the library's own .gitignore files (at any level under the root; global git excludes and .git/info/exclude are not read), by an optional .markindexignore file at the root (same syntax, for rules you don't want in .gitignore) and by --ignore options (gitignore syntax, relative to the root, repeatable; ignore: string[] in the library API; also accepted by mcp --reindex-on-start; a leading ./ is dropped, a leading / anchors at the root as in gitignore, and absolute filesystem paths or ../ are rejected). The sources only add exclusions: a ! rule in .markindexignore or --ignore cannot re-include a path that a .gitignore excludes. A document that becomes ignored is removed from the index on the next run. Dot-directories such as .git and .markindex are skipped and symlinks are not followed. One writer at a time (a second index fails fast); readers such as the MCP server are never blocked. When the index was built by an older MarkIndex version or with different tree parameters, the next index run rebuilds every document (full rebuild: … in the output, rebuilt in the report); cached summaries are reused.

Page breaks from PDF / HTML converters. Converters often emit a bare --- line right after a page's last paragraph. CommonMark reads that as a setext heading, so the whole paragraph becomes a level-2 heading. index warns when it sees setext h2 headings whose text spans several lines or exceeds 120 characters (warning : … on stdout, warnings in the JSON report). For such libraries pass --no-setext (also on mcp --reindex-on-start; setext: false in the library API): --- then is a thematic break and === plain text. The setting is stored in the index: later runs without the flag (index, mcp --reindex-on-start, Library.index()) keep it; pass --setext (setext: true) to turn it back on. Changing it rebuilds the index.

Library usage

import { openLibrary } from 'markindex'

// Index (or update) a folder, then open it read-only for agents. const writer = openLibrary({ root: './docs', readonly: false }) await writer.index() // { added, updated, removed, unchanged, skipped, summaryErrors, warnings } writer.close()

const lib = openLibrary({ root: './docs' }) const envelope = await lib.callTool('search_documents', { query: 'termination notice period' }) console.log(JSON.parse(envelope)) // { success: true, data: { results: [...] }, next_steps: {...} }

With the Vercel AI SDK (your agent drives the tools)

import { generateText, stepCountIs } from 'ai'

import { anthropic } from '@ai-sdk/anthropic' import { AGENT_INSTRUCTIONS, createAiSdkTools, openLibrary } from 'markindex'

const lib = openLibrary({ root: './docs' }) const { text } = await generateText({ model: anthropic('claude-sonnet-5-5'), system: AGENT_INSTRUCTIONS, // browse first, read structure, cite [doc_path, L - ] tools: createAiSdkTools(lib), stopWhen: stepCountIs(15), prompt: 'What notice period applies when an employee resigns?', })

Built-in evidence retrieval

retrieve() runs the same tools in a model loop and returns evidence only (original text re-read from the index by line range — never model-written text), plus usage and why it stopped:

import { anthropic } from '@ai-sdk/anthropic'

import { openLibrary, retrieve } from 'markindex'

const lib = openLibrary({ root: './docs' }) const { evidence, usage, stopReason } = await retrieve(lib, 'What notice period applies when an employee resigns?', { model: anthropic('claude-sonnet-5-5'), maxSteps: 12, maxInputTokens: 200_000, rateLimit: { requestsPerMinute: 50, inputTokensPerMinute: 40_000 }, // optional, client-side }) for (const e of evidence) console.log(${e.docPath} L${e.startLine}-${e.endLine}, e.titlePath.join(' > '))

Lower-level building blocks are exported too: parseMarkdown, buildTree, optimizeTree, openStore, indexLibrary, searchNodes, summarizeDocument, estimateSummaryCost, TOOL_CONTRACT, createMcpServer.

MCP setup

Build the index first (npx markindex index /path/to/docs), then register the server. MarkIndex is also listed in the official MCP Registry as io.github.jackjin1997/markindex, so registry-aware clients can install it from there (they start it as npx markindex mcp --reindex-on-start).

Claude Code

claude mcp add markindex -- npx markindex mcp /path/to/docs

Cursor (~/.cursor/mcp.json or .cursor/mcp.json in a project)

{

"mcpServers": { "markindex": { "command": "npx", "args": ["markindex", "mcp", "/path/to/docs"] } } }

The server sends AGENT_INSTRUCTIONS as its MCP instructions and marks all tools read-only.

Tools

| get_document_structure | The section outline (node ids, titles, line ranges, tokens, summaries