Code Crawler

Code-Crawler

Static + dynamic call tracing MCP server backed by a persistent property graph of your codebase.

Code-Crawler indexes a project into a SQLite-backed property graph — files, packages, modules, classes, interfaces, methods, fields, imports, annotations/decorators, call edges, virtual dispatch, aspect advice, and the wiring frameworks add (entry points, routes, injected dependencies, reflective lookups) — then exposes that graph to LLM clients as MCP tools: find callers, trace call paths, explore neighborhoods, follow runtime dispatch, and search symbols across the whole codebase.

Supports Java, Python, TypeScript, and JavaScript projects, and can index all of them into the same graph.

📖 Documentation site: https://nithin554.github.io/code-crawler/ 📘 Task-oriented guide: USAGE.md 🗺️ Roadmap: ROADMAP.md

        ┌───────────────────────  MCP (stdio)  ───────────────────────┐

│ crawl_project search_symbols get_callers get_callees │ │ trace_path trace_execution find_aspects get_decorators│ │ find_implementations find_hotspots find_dead_code ... │ └───────────────┬──────────────────────────────┬───────────────┘ │ │ ┌──────▼───────┐ ┌────────▼────────┐ │ crawler │ │ graph queries │ │ parse+resolve│ │ recursive CTEs │ └──────┬───────┘ └────────┬────────┘ │ │ ┌──────▼──────────────────────────────▼──────┐ │ SQLite property graph (WAL) │ │ nodes (kind, fqn, props) + edges (kind) │ └─────────────────────────────────────────────┘

Why a graph?

Plain-text grep can't answer questions like:

• *"Who transitively calls OrderRepository#create?"*
• *"Is there a static path from Main#main to Tracking#dispatch?"*
• *"What is the impact radius of changing OrderService#place?"*

These need graph traversal. Code-Crawler builds the graph once (idempotently, file-by-file) and answers them with bounded recursive SQL queries.

Features

• Four languages, one graph: Java (tree-sitter-java), Python (tree-sitter-python), TypeScript (tree-sitter-typescript, tree-sitter-tsx), and JavaScript (tree-sitter-javascript) — all WASM, no native build step. Language is detected per file, so a polyglot repo lands in a single graph.
• Dynamic tracing on a static graph — DYNAMIC_DISPATCH edges connect interface/base calls to every override in the project, and ADVISES edges weave AspectJ/Spring AOP advice into execution traces (before, after, around, after-returning, after-throwing, named @Pointcut references, &&/||/! and execution, within, @annotation, @within, target, this, args primitives).
• Run-time indirection is named, not lost — Java reflection/ServiceLoader, Python getattr/importlib, and JS require/dynamic import() produce REFLECTS edges to the resolved symbol when the target is statically determinable, or to a __reflect__: marker when it is not.
• Framework wiring — dependency injection (@Autowired, NestJS constructor injection, FastAPI Depends) becomes INJECTS edges, and HTTP routes (@GetMapping, @RequestMapping, NestJS @Controller/@Get, FastAPI/Flask/Quart decorators, Express app.get) become ROUTES edges to the handler, which is also marked as an entry point.
• Control flow on call edges — every CALLS edge records whether the call is conditional and the compact source text of its enclosing guards/loops, so an agent can tell a happy path from an error path without reading the file.
• External dependencies are explicit — third-party imports and runtime globals become package-level external_symbol nodes with REFERENCES edges and a find_external_dependencies summary, distinct from project code.
• Annotations as first-class nodes — @Transactional, @Cacheable, @Async, @Retryable, Python decorators, and TS decorators become annotation nodes with ANNOTATED_WITH edges, so you can query decorators on types, methods, and fields.
• Call resolution heuristics that go beyond naive string matching: receiver types inferred from locals, parameters, fields, new T() expressions, self/this scoping, dotted paths (service.repo.find), loop-variable element types, bare/this/super calls, static/qualified calls, and superclass chains. Unresolvable calls become unknown_symbol phantom nodes with resolved:false edges, so the graph stays traversable.
• Incremental, file-scoped indexing — each crawl records a per-file manifest (size, mtime, SHA-256) and replays the cached parse result for files whose content is unchanged, so a re-crawl re-parses only what actually changed. Changes are reported as filesAdded / filesUpdated / filesUnchanged / filesRemoved. Derived dynamic edges and orphan pruning are rebuilt only when something changed. A full re-crawl still removes files that disappeared, without duplicating the graph, and scoped crawls (fileFilter or a forced language) never remove files outside their scope. When a checkout, rsync -t, an mtime-preserving unzip, or clock skew restores an edit with the old size and mtime, the shortcut misses it — pass force (--force on the CLI) to re-parse everything.
• FTS5-backed symbol search — symbol lookup runs against an external-content FTS5 index with BM25 ranking and exact/prefix boosting, falling back to a LIKE scan only when FTS finds nothing. Search does not degrade as the graph grows.
• Versioned schema with automatic migration — the database carries a schema version and upgrades itself in place on open. A database written by a newer build is rejected with a clear message instead of being read as if the shape were known.
• A database per project — the index lives in /.code-crawler/index.db, so two repositories can never collide. Pointing a populated database at a different project fails with a clear_index hint instead of silently wiping the first index.
• Analytics — hotspots (fan-in/fan-out/advice counts) and dead-code candidates.
• Self-contained: single SQLite file, no external services.
• LLM-friendly: every tool returns structured JSON, unknown symbols come back with fuzzy search suggestions.

Install & build

Requires Node.js ≥ 22.12 (better-sqlite3@13 and the Vitest toolchain both need it).

npm install

npm run build # tsc → dist/ npm test # vitest (full suite; set CODECRAWLER_BENCH=1 for the scale test) npm run lint # biome check (lint + format, fails CI) npm run format # biome format --write npm run coverage # vitest run --coverage (thresholds in vitest.config.ts) npm run bench # crawl/query benchmark on a generated ~137k-line project

CLI

codecrawler crawl <path>     # index a project

codecrawler crawl --force # re-parse every file, ignoring cached metadata codecrawler status # index summary codecrawler query # fuzzy-lookup a symbol codecrawler serve # start the MCP server over stdio (default)

Environment variables:

Invalid values never crash the server: booleans fall back to their default and integers are clamped to their documented range, each with a one-line warning on stderr.

What gets indexed

crawl walks the project and skips .git, node_modules, build output, and other CODECRAWLER_EXCLUDE_DIRS entries. On top of that:

• the project's `.gitignore` is honored (disable with CODECRAWLER_RESPECT_GITIGNORE=false),
• an optional `.crawlerrc` in the project root can narrow or widen discovery with gitignore-style globs, and relocate where module ids start:

  {

"include": ["src/", "packages/core/"], "exclude": ["**/generated/"], "moduleRoots": ["src"] }

When include is non-empty, only matching files are indexed. A malformed .crawlerrc is ignored with a warning rather than failing the crawl.

moduleRoots is opt-in: with ["src"], src/orders/service.ts becomes the module orders.service instead of src.orders.service, which is what a baseUrl-or-PYTHONPATH project actually imports. It is never inferred, because inferring it would silently renumber an existing index. TypeScript and JavaScript projects also get path aliases for free: compilerOptions.paths and baseUrl are read from tsconfig.json (or jsconfig.json), following extends within the project, so import { x } from "@app/orders/service" resolves to the real module instead of being recorded as external. Changing either setting re-parses cached files on the next crawl.

• files above CODECRAWLER_MAX_FILE_BYTES and files that look binary are skipped; crawl_project reports them in skippedFiles with a reason.
• per-file parse failures never abort the crawl; they are reported in parseErrors.

Only the root .gitignore is read — nested .gitignore files are not yet supported.

Incremental crawls

A crawl records a manifest row per file (size, mtime, SHA-256, language). On the next crawl a file whose size and truncated mtime both match is served from a cached parse result instead of being re-read and re-parsed; a file whose mtime changed but whose hash did not is also replayed from cache. Resolution, phantom-node creation and derived edges still run over the whole in-memory file set, so cross-file edges stay correct — only I/O, parsing and database writes are skipped.

crawl_project reports the split:

Known limitation: because an unchanged file's edges are not rewritten, a call that becomes *newly resolvable* only because some other file changed (for example a method gained an ove