Context Mem

Context Mem

Memory + context infrastructure for AI agents. Remembers everything. Compresses everything. Fully local.

-gold)

---

The Problem

Two problems with today's AI tooling that no one has solved together in a single package.

Your AI forgets. Every new session starts from zero. The architecture decisions you settled on last Thursday, the bug you spent four hours tracing to a misconfigured environment variable, the preferences you stated three times — none of it carries forward. You spend the first ten minutes of every session re-explaining context that already existed. Multiply this by every developer on your team, every project, every day.

Your context explodes. Long coding sessions blow past the context window. A typical session with 50 tool outputs accumulates 365 KB of raw text — stack traces, test output, file reads, shell commands. Every token costs money or slows the model. Naive truncation drops the exact evidence the model needs. Keeping everything makes responses slower and inference cost climb fast.

These two problems compound each other. The solution to forgetting (keep everything) is the opposite of the solution to context explosion (discard everything). The result is a false tradeoff most tools force on you: either your AI forgets everything, or your costs balloon. context-mem solves both simultaneously by building an indexed, compressed, retrievable memory store rather than dumping raw history into the context window.

---

The Solution — one tool, two pillars

Pillar 1: Memory (LLM Wiki)

Every tool call is automatically ingested, summarized, and written into a navigable markdown vault — a living wiki your AI maintains about your project. Entities get their own pages with backlinks. Topics get synthesis pages. Sessions become browseable source documents. Decisions accumulate into a reconstructible trail.

The vault lives at .context-mem/vault/ and syncs continuously from the underlying SQLite store. Read it in Obsidian, grep it from the terminal, or query it through 45+ MCP tools using hybrid BM25 + vector + optional LLM judge search. The raw SQLite store is the authoritative record; the markdown vault is the derived, human-readable layer.

This is a reference implementation of Andrej Karpathy's LLM Wiki pattern — three layers (raw sources / wiki / schema), with automatic ingest from tool calls that no other system provides.

Pillar 2: Compression (14 summarizers)

Every observation passes through a content-aware summarizer before storage. A stack trace is not treated the same way as a JSON config file. Shell output from a build is compressed differently from TypeScript compiler errors. The system applies the right compression for the content type.

The result: a full coding session with 50 tool outputs goes from 365 KB to 3.2 KB — 99.1% token savings, verified. Compression is adaptive: recent high-importance observations stay verbatim; older low-importance ones compress progressively. Pinned entries never compress regardless of age.

---

One Command

npm i context-mem && npx context-mem init

init auto-detects your editor and writes the right config files:

No API keys. No cloud account. No data leaves your machine.

---

Dual-pillar in 60 seconds

[ placeholder: GIF or video — Claude Code session with split view showing Obsidian graph updating in real time alongside the context-mem dashboard token savings chart ]

---

Architecture (reference implementation of Karpathy's LLM Wiki pattern)

                    ┌─────────────────────────────────────────┐

│ Raw Sources (immutable) │ │ tool calls · observations · file reads │ └──────────────────┬──────────────────────┘ │ ▼ ┌─────────────────────────────────────────┐ │ Observation Pipeline │ │ │ │ PrivacyEngine (9 detectors) │ │ → 14 content-aware summarizers │ │ → entity extraction (100+ aliases) │ │ → topic detection │ │ → importance scoring (0.0–1.0) │ │ → adaptive compression tier │ └────────────────┬────────────────────────┘ │ ┌─────────────────┴───────────────────┐ │ │ ▼ ▼ ┌──────────────────────────┐ ┌─────────────────────────────┐ │ SQLite (primary) │ │ Markdown Vault (derived) │ │ │ │ │ │ observations │──────▶│ .context-mem/vault/ │ │ entities + graph │ sync │ index.md │ │ knowledge │ │ log.md │ │ events │ │ sources/ .md │ │ FTS5 index │ │ entities/ .md │ │ vector embeddings │ │ topics/ .md │ └──────────────────────────┘ │ knowledge/ .md │ │ └─────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────────────┐ │ Hybrid Retrieval │ │ │ │ BM25 (8 strategies + synonym expansion) │ │ + Vector (nomic-embed-text-v1.5, 768-dim) │ │ + Trigram + Levenshtein │ │ → Fusion (intent-adaptive weights, IDF reranker) │ │ → Optional LLM judge (Haiku, 50/50 blend, 100% R@5) │ └──────────────────────────────────────────────────────────────┘

Three layers (per Karpathy):

• Raw sources — your tool call outputs, file reads, shell commands, observations. Written once, never modified. The permanent record.
• The wiki — LLM-maintained markdown vault (.context-mem/vault/). Auto-synced from SQLite. Human-readable, Obsidian-compatible, grep-friendly. Entity pages, topic pages, session pages, knowledge pages, index, event log.
• Schema — docs/llm-wiki-schema.md governs page structure, linking conventions, agent workflow recipes, and interop contract. Public spec — other tools can emit conforming wikis.

The distinction from most memory systems: context-mem is not replacing SQLite with markdown. SQLite is authoritative — it is where observations are stored, searched, and indexed. The vault is the browseable, linkable, diffable surface on top of it — the layer a human or LLM can navigate without a database client. If you delete the vault directory, you lose nothing that matters. If you edit a vault page manually, those edits are preserved and not overwritten on the next sync.

This is the Karpathy three-layer model applied to a running AI development environment: immutable inputs, a maintained synthesis layer, and a public schema that governs the synthesis. The vault can be used independently of the MCP tools — it is just a directory of markdown files. Open it in any editor. Put it in git. Diff it across commits. Use it as long-form context by copy-pasting pages into a new conversation. The MCP tools are the automated path; the markdown vault is the portable, durable, human-readable path.

---

Retrieval benchmarks (honest methodology)

All scores are session-level retrieval recall: did any correct evidence session appear in the top-k results? This is different from end-to-end QA accuracy (retrieve + generate + judge), which is harder and lower for every system. Both measurements are published here.

Pure local (zero API calls, fully free)

With optional LLM reranking (~$1 per 500 queries)

The LLM judge (Claude Haiku) scores the top-N BM25+vector candidates 0–10 and blends 50/50 with the retrieval score. Activates when ai_curation.enabled = true. Adds ~$0.002 per query at Haiku pricing.

Methodology notes:

• A "hit" is scored if any correct evidence session appears in top-k. Not end-to-end QA.
• LoCoMo benchmark appends dataset-provided metadata (session_summary, observation, event_summary) to session documents — the production system applies equivalent enrichment via summarizers and entity extraction.
• Synonym expansions: core query-builder includes general-vocabulary synonyms (movie → film, sibling → brother). Results without any synonym expansion are ~1-2% lower.
• All benchmark code is open and runnable: npm run bench. See benchmarks/.

_Full methodology: docs/benchmarks/methodology.md (published with v3.4)._

---

Compression benchmarks (verified)

Per-summarizer breakdown:

Compression is lossless at the semantic level for high-importance observations (DECISION, MILESTONE, PROBLEM flags) — those stay verbatim regardless of age. Compression applies to routine tool output.

---

Core features

Memory

• LLM Wiki substrate — markdown vault at .context-mem/vault/, auto-synced from SQLite. Entity pages, topic pages, session source pages, knowledge pages, index.md, log.md. Obsidian-compatible, grep-friendly.
• **14 content