Document corpus

mdcx

Convert a document collection to verified Markdown, package it into a single encrypted archive with its search index and provenance, and serve it to agents over the Model Context Protocol.

Contents

• Scope
• Requirements
• Installation
• Quick start
• Conversion
• Packaging
• Querying
• Retrieval model
• Calibration
• Vocabulary
• Transcription recovery
• Expressions
• Quotations across a cut
• Attachments
• Dates
• Correspondence
• Incremental packaging
• Resident conversion
• MCP server
• Sources
• Portable paths
• Signing
• Encryption
• Package format
• Limitations
• Tests
• Contributing
• Security
• Releases
• Authorship
• Citation
• Licence

Scope

mdcx addresses one constraint. An agent asked a question about a document collection must either receive the documents in its context window, which is bounded and billed per token, or query a component that holds an index and returns only the passages that bear on the question. mdcx implements the second.

Three properties define the result:

• Fidelity is checked. Each conversion is compared against the text the original exposes and the coverage achieved is recorded per file. Files that expose no text are marked unverifiable rather than reported as complete. What the comparison establishes differs by engine, and Verification states which.
• The corpus is one artefact. Passages, index and provenance are held in a single AES-256-GCM file whose header can be read without the key, and which may be signed.
• Answers are citable. Every passage carries its source document and its position, so an answer can be quoted against a location.

The pipeline has three stages. Conversion attempts each document with the least expensive engine able to read it and escalates only where that engine falls short: direct text extraction, then a pass that recovers drawn tables, then layout analysis, and optical character recognition for documents that expose no text. Packaging writes the corpus, its index and the provenance of each passage to one file. Retrieval merges word matching and dense retrieval by reciprocal rank, and states when nothing in the corpus bears on the question rather than returning its nearest passage.

Requirements

Python 3.11 or later. No other component is required to query a package.

The floor is 3.11 because a package is one SQLite database, loaded from the decrypted bytes when it is opened, and sqlite3 gained the required call in that version. Conversion and cross-language retrieval each add dependencies, listed under Installation.

Installation

Querying and conversion are separated because their requirements differ by roughly two orders of magnitude. A recipient who only reads packages installs neither Docling nor PyTorch.

The multilingual extra supplies the embedding model, downloaded once on first use, and is required only for queries that cross languages.

The tables extra covers tables that are not drawn with rules — a screenshot of a spreadsheet, or a layout held together by alignment. It reads the geometry only; cell contents still come from the text layer of the document. Without it those pages are handled by the layout engine.

Systems with a CUDA device

Install mdcx[all-gpu] rather than mdcx[all], and install onnxruntime-gpu separately.

onnxruntime and onnxruntime-gpu publish the same module and cannot coexist: whichever was installed last takes effect. An extra that pins the CPU build will therefore displace an accelerated one on every upgrade. all-gpu is all without that pin.

pip cannot express a dependency satisfied by either distribution, so this cannot be settled by declaration alone. mdcx-convert checks at startup and reports when the machine has a device the runtime does not offer. The check on its own:

python -c "import onnxruntime as o; assert 'CUDAExecutionProvider' in o.get_available_providers()"

Quick start

pip install "mdcx[convert]"

mdcx-convert --input ./Documents --output ./Documents_md mdcx pack --output ./Documents_md --target corpus.mdcx --key "passphrase" mdcx search corpus.mdcx "where is the storage temperature stated" --key "passphrase"

Conversion

mdcx-convert --input ./Documents --output ./Documents_md

The output folder mirrors the input directory structure, one .md per source document, together with an index of the run.

An interrupted run resumes: work already recorded is not repeated unless --force is given.

Long documents are split into chapters, each converted independently, with an index document recording the correspondence to the original. A chapter is converted from an extract of the document, and its page markers state the document's own numbering rather than the extract's; its front matter declares first_page and last_page, so a passage can be cited back to the page it came from. The index document declares type: document_index and holds no text of its own — the text is in the chapters. --sample-pages converts a spread sample instead of the whole document; the front matter records both the pages taken and the document's total, so a sample is not mistaken for a short document.

--timeout exists because some documents do not complete under layout analysis. Without a limit one such document holds a worker for the length of the run. --recycle-after replaces a worker periodically, since abandoning a document does not stop the thread it started.

Formulas

The text layer of a PDF does not carry the structure of a formula: the bar of a fraction is a drawn stroke, and a superscript is loose text on another line. A faithful extraction of that layer therefore renders a quotient as a single line. Nothing has gone wrong — the structure was never there to extract.

--formulas transcribes the formulas to LaTeX, which means reading their image rather than their text. That is recognition rather than extraction: a different kind of claim about the document, and one that can be wrong in ways extraction cannot, so it is asked for rather than assumed. It needs the structured engine — --no-docling turns it off and the run says so — and a model that is fetched once with docling-tools models download-hf-repo docling-project/CodeFormulaV2; where the model is missing the run refuses at the start rather than failing on the first page that holds a formula.

It is not free. Measured on one 20-page paper on this machine: 17.7 s without and 348.7 s with, for 56 transcribed formulas where the text layer yielded none. Prose gains nothing from it.

The transcription is appended under its own heading rather than replacing the text. On that paper the structured engine transcribed the formulas and lost 11% of the words, while native extraction covered the text and carried no formula — so choosing between them gives up one of the two. The record reports how many were transcribed as formulas.

Verification

Each conversion is compared against the text the original exposes. The index records the coverage achieved per file. Documents that expose no text — scanned drawings, for example — are marked unverifiable, since no text original exists to measure against.

The reference is read by a library independent of the layout and OCR engines, so for those the comparison is between two different readings of the document. For native extraction it is not: the reference and the extraction are the same read, and the comparison measures what the subsequent structuring and compaction kept. The reference is read once and used for both.

Packaging

mdcx pack --output ./Documents_md --target corpus.mdcx --key "passphrase"

--output accepts a folder of Markdown, a single file, or a .jsonl file with one record per line, which is how a catalogue of records is packed without first writing it to disk.

--fast selects a lower compression preset, and --preset names one directly. Sealing is a fixed price paid on every write: a package meant to be distributed is compressed once and downloaded many times, while a corpus rebuilt whenever it grows pays the clock and not the bytes. The level travels inside the compressed stream, so a package written at any level is opened by any reader, and nothing else about the package changes.

--compression chooses what compresses the body. The header records the name and a package is read by what its header names, so existing packages are unaffected and either kind is opened by a reader that has the module for it. LZMA is the default and needs nothing; zstd needs mdcx[zstd]. Which is smaller depends on the material — measured on one corpus zstd was small