Rag

FieldCure MCP RAG Server

A Model Context Protocol (MCP) server for indexing and searching local document collections. Supports DOCX, HWPX, PDF (with OCR), Excel, PowerPoint, and audio (Whisper transcription, Windows-only), with hybrid keyword + semantic search optimized for Korean and English.

Built with C# and the official MCP C# SDK.

Commands

fieldcure-mcp-rag

├── serve --base-path # Multi-KB MCP search server (stdio) ├── exec --path [--force] [--partial ...] # Headless indexing for a single KB ├── exec-queue --queue-file [--sweep-all] # Process deferred indexing queue ├── prune-orphans --base-path # Delete orphan KB folders └── smoke-ocr --pdf # Self-test: OCR a scanned PDF (Windows)

• serve — read-only MCP server serving all knowledge bases under the base path. Single process handles multiple KBs via kb_id parameter. Can run while exec is indexing (SQLite WAL).
• exec — scans source folders, chunks documents, contextualizes with AI, embeds, stores in SQLite. --partial re-runs only downstream stages when models change, preserving OCR output.
• exec-queue — sequential orchestrator consuming a deferred indexing queue. One entry at a time, no GPU contention. --sweep-all processes deferred entries too (used at app shutdown).
• prune-orphans — deletes orphan KB folders (GUID-named, no config.json). Protected folders (., _ prefix, -backup-) are never touched.
• smoke-ocr — diagnostic mode. Loads a scanned PDF through the OCR fallback parser, prints recognized text to stdout, and exits 0 on a non-empty result. Surfaces DllNotFoundException / BadImageFormatException distinctly so a missing or arch-mismatched native is immediately visible. Useful for verifying that the OCR native path is wired correctly on a given host (notably win-arm64 dnx installs).

Features

Search

• Hybrid BM25 + vector search with Reciprocal Rank Fusion (RRF)
• BM25-only fallback when no embedding provider is configured
• Korean-optimized chunking (sentence boundary, decimal protection, parenthesis-aware)
• SIMD-accelerated cosine similarity via System.Numerics.Vector
• FTS5 trigram index for substring and CJK-friendly keyword matching

Indexing

• Incremental indexing with SHA256 change detection
• AI-powered chunk contextualization with bilingual keyword enrichment (see Chunk Contextualization)
• 2-commit pipeline preserves expensive upstream work across embedding failures (see How Indexing Works)
• Math equation extraction from DOCX/HWPX as [math: LaTeX] blocks
• PDF with OCR fallback (Tesseract eng+kor) for scanned pages
• Audio transcription (.mp3, .wav, .m4a, .ogg, .flac, .webm) via Whisper.net — Windows-only. Model size (Tiny→Large) is auto-selected from detected GPU/RAM/cores at startup; each transcript chunk records audio.model_size and audio.transcribed_at for future reindex auditing
• Cross-process indexing lock with stale PID auto-cleanup
• Orphan cleanup for deleted files

Queue Orchestrator

• All indexing requests flow through start_reindex MCP tool — no direct exec spawn
• Scope merge rules: full ⊃ contextualization ⊃ embedding (duplicate requests upgrade, not duplicate)
• PID-based orchestrator lock with reuse defense (orchestrator.lock)
• Logical KB deletion (config.json removal) + prune-orphans physical cleanup
• Deferred indexing for app-shutdown batch processing (--sweep-all)

Operations

• Multi-KB serve: single process serves all knowledge bases under a base path, lazy-loaded per KB
• SQLite WAL mode allows search during indexing
• Graceful shutdown via cancel file
• Per-KB config.json with provider configuration

Integration

• Ollama native — embedding via /api/embed, contextualization via /api/chat with keep_alive and num_ctx support. Requires Ollama 0.4.0+.
• OpenAI-compatible — embedding via /v1/embeddings, contextualization via /v1/chat/completions. Works with OpenAI, Azure OpenAI, Groq, LM Studio, Together AI.
• Gemini native — embedding via /v1beta/models/{model}:embedContent with task_type asymmetric retrieval (RETRIEVAL_DOCUMENT / RETRIEVAL_QUERY) and Matryoshka dimension truncation (768 / 1536 / 3072). gemini-embedding-2, multilingual, 8k token input.
• Anthropic — contextualization via /v1/messages.
• API keys via environment variables — OPENAI_API_KEY, ANTHROPIC_API_KEY, etc. Batch indexing commands (exec, exec-queue) are env-var-only. Interactive MCP search can fall back to MCP elicitation when the client supports it.
• Standard MCP stdio transport (JSON-RPC over stdin/stdout)

Chunk Contextualization

Standard RAG chunking loses context — a sentence about "the protocol" becomes ambiguous when ripped from its surrounding paragraphs. This server addresses that with Unified Chunk Contextualization: a single LLM call per chunk that produces both contextual framing and bilingual (Korean + English) keywords in one pass.

The result is stored alongside the original chunk text:

• Original text is preserved for accurate retrieval display
• Contextualized text is what gets embedded and indexed in BM25
• Bilingual keywords enable cross-lingual search — a Korean query can retrieve English documents and vice versa

This is enabled by setting contextualizer in config.json. It can be disabled (set provider/model to empty) if you prefer raw chunk indexing.

How Indexing Works

The exec command runs a 5-stage pipeline per file:

1. Extract — text from document (DOCX, PDF OCR, audio transcription, etc.)
2. Chunk — split into ~1000 char windows
3. Contextualize — LLM enrichment (optional, see above)
4. Embed — vector embedding via API
5. Persist — save to SQLite

For large files, Stage 1 alone can take 20+ minutes — OCR on a 596-page scanned PDF, or Whisper transcription of a multi-hour audio recording. The first audio file in any KB also pays a one-time ggml model download (cached under {UserProfile}/.fieldcure/whisper-models/). To prevent expensive upstream work from being lost when later stages fail, the pipeline uses a 2-commit model:

Stages 1-3 (Extract → Chunk → Contextualize)

↓ [Commit 1] chunks saved as PendingEmbedding ↓ Stage 4 (Embed) ├─ success → [Commit 2a] promote chunks to Indexed └─ failure → chunks remain PendingEmbedding (retry next exec)

Why this matters: A 25-minute OCR result is persisted on disk before any embedding API call. If Stage 4 fails (network error, rate limit, token limit, process crash, even power loss), the chunks survive. The next exec hash-skips the file (no OCR re-run) and the deferred retry pass attempts only Stage 4.

Per-Chunk Failure Isolation (Binary Split)

If a single chunk in a file exceeds the embedding model's token limit (e.g., a math-dense page in a textbook), the binary split algorithm isolates that one chunk:

EmbedBatch([0..1249])         → 400 "input[846] too long"

├─ EmbedBatch([0..624]) → OK (promote 625) └─ EmbedBatch([625..1249]) → 400 ├─ EmbedBatch([625..937]) → 400 │ ... (binary search narrows toward chunk 846) │ └─ EmbedBatch([846..846]) → 400 (mark chunk 846 Failed) └─ EmbedBatch([938..1249]) → OK (promote 312)

Result: 1249 chunks indexed, only chunk 846 marked Failed. The file's status becomes Degraded — partially searchable instead of completely missing.

Deferred Retry Pass

Each exec ends with a retry pass over any chunks left in PendingEmbedding state from previous runs:

• Reads enriched text from DB — no OCR or contextualization re-run
• Calls the embedding API only — typically seconds, not minutes
• Up to 3 retries per chunk; on exhaustion, the chunk is marked Failed
• Auth errors (401/403) flag the provider as unavailable and skip the rest of the pass

File States

Schema Versioning

Each KB DB carries a PRAGMA user_version tag. The exec command migrates older schemas automatically as part of InitializeSchema(). The serve command opens DBs read-only and never triggers migration — older-schema KBs continue to serve search queries correctly while their new-feature columns remain unused.

Installation

dotnet tool (recommended)

dotnet tool install -g FieldCure.Mcp.Rag

From source

git clone https://github.com/fieldcure/fieldcure-mcp-rag.git

cd fieldcure-mcp-rag dotnet build

Requirements

• .NET 8.0 Runtime or later
• OCR: Windows x64 only — Tesseract OCR for scanned PDFs loads lazily on first use (Windows only). On other platforms, PDFs with embedded text work normally; scanned pages without a text layer are silently skipped.
• An embedding provider (Ollama, OpenAI, etc.) — optional, BM25 search works without it
• Ollama 0.4.0 or later (if using Ollama for embedding or contextualization)

Quick Start

Index a folder and search it without any embedding setup (BM25 only):

# 1. Install

dotnet tool install -g FieldCure.Mcp.Rag

2. Create a minimal config

$kbPath = "$env:LOCALAPPDATA\FieldCure\Mcp.Rag\demo" New-Item -ItemType Directory -Force -Path $kbPath @' { "id": "demo", "name": "Demo KB", "sourcePaths": ["C:\\my-docs"] } '@ | Set-Content "$kbPath\config.json"

3. Index

fieldcure-mcp-rag exec --path $kbPath

4. Start the search server

fieldcure-mcp-rag serve --base-path "$env:LOCALAPPDATA\FieldCure\Mcp.Rag"

For full retrieval quality with semantic search and contextualization, add embedding and contextualizer blocks to config.json — see Usage below.

Usage

1. Create a knowledge base folder

%LOCALAPPDATA%\FieldCure\Mcp.Rag\{kb-id}\config.json

{

"id": "my-kb-001", "name": "Project Docs", "created": "2026-04-03T00:00:00Z", "sourcePaths": ["C:\\Users\\me\\Documents\\project-docs"], "contextualizer": { "provider": "anthropic", "model": "claude-haiku-4-5-20251001", "apiKeyPreset": "Claude" }, "embedding": { "provider": "openai", "model": "text-embedding-3-small", "apiKeyPreset": "OpenAI" } }

API keys are resolved from environment variables: apiKeyPreset: "OpenAI" → OPENAI_API_KEY, "Claude" → ANTHROPIC_API_KEY, "Gemini" (or "Google") → GEMINI_API_KEY.

Gemini embedding example — asymmetric retrieval with 1536-dim Matryoshka truncation (50% storage of full 3072 with identical MTEB score):

"embedding": {

"provider": "gemini", "model": "gemini-embedding-2", "apiKeyPreset": "Gemini", "dimension": 1536 }

In serve mode, `search_d