ShadowCrawl

CortexScout (cortex-scout) — Search and Web Extraction Engine for AI Agents

CortexScout is the Deep Research & Web Extraction module within the Cortex-Works ecosystem.

Designed for agent workloads that require token-efficient web retrieval, reliable anti-bot handling, and optional Human-in-the-Loop (HITL) fallback.

---

Overview

CortexScout provides a single, self-hostable Rust binary that exposes search, extraction, and stateful browser automation capabilities over MCP (stdio) and an optional HTTP server. Output formats are structured and optimized for downstream LLM use.

It is built to handle the practical failure modes of web retrieval (rate limits, bot challenges, JavaScript-heavy pages) through progressive fallbacks: native retrieval → Chromium CDP rendering → Stateful E2E Testing → HITL workflows.

---

Tools (Capability Roster)

Legacy names remain callable as compatibility aliases (web_search_json, web_fetch_batch, web_crawl, fetch_then_extract, human_auth_session). Agents should prefer the unified primary tools above.

---

Ecosystem Integration

While CortexScout runs as a standalone tool today, it is designed to integrate with CortexDB and CortexStudio for multi-agent scaling, shared retrieval artifacts, and centralized governance.

---

🎭 The "Playwright Killer" (Stateful Browser Automation)

CortexScout includes a built-in, stateful CDP automation engine designed specifically for AI Agents, completely replacing heavy frameworks like Playwright or Cypress for E2E testing workflows.

• The Silent Omni-Tool (`scout_browser_automate`): Instead of calling dozens of browser tools, agents pass one array of steps. The runtime now covers Playwright-style action families in one call: navigation, hover/click/type/wait, locator-driven actions, assertions, tabs, screenshots/PDF, file upload, form fill, dialog policy, coordinate mouse actions, route mocking, console/network capture, and cookie/storage CRUD.
• Persistent Agent Profile: Automation runs silently in the background (--headless=new) using a dedicated isolated profile (~/.cortex-scout/agent_profile). It maintains cookies, localStorage, and session state across tool calls without causing SingletonLock collisions with your active desktop browser.
• QA Mock, Trace, And Verification Engine: Agents can install route mocks (mock_api, route_list, unroute) with response header overrides/stripping, trace flows (trace_start, trace_stop, trace_export), capture console/network logs, checkpoint browser state, and run both CSS and locator-based assertions plus Playwright-style verification helpers.
• The Agent Auth Portal (`scout_agent_profile_auth`): If the silent agent encounters a CAPTCHA or complex OAuth login (like Google/Microsoft) on a new domain, this tool launches the agent's profile in a visible window. You solve the CAPTCHA once, the cookies are saved, and the agent returns to silent automation forever.

Playwright-Style Coverage Map

The main tradeoff versus raw Playwright MCP is packaging, not capability shape: Cortex Scout keeps the browser surface inside one stateful omni-tool so agents spend fewer turns and fewer tokens coordinating multi-step flows.

---

Anti-Bot Efficacy & Validation

This repository includes captured evidence artifacts that validate extraction and HITL flows against representative protected targets.

See proof/README.md for methodology and raw outputs.

---

Quick Start

Option A — Prebuilt binaries

Download the latest release assets from GitHub Releases and run one of:

• cortex-scout-mcp — MCP stdio server (recommended for VS Code / Cursor / Claude Desktop)
• cortex-scout — optional HTTP server (default port 5000; override via --port, PORT, or CORTEX_SCOUT_PORT)

Health check (HTTP server):

./cortex-scout --port 5000

curl http://localhost:5000/health

Option B — Build from source

Install protoc first. lance-encoding uses Protocol Buffers during the release build, so protoc must be on your PATH.

• macOS: brew install protobuf
• Ubuntu/Debian: sudo apt-get install -y protobuf-compiler
• Fedora: sudo dnf install -y protobuf-compiler

Basic build (search, scrape, deep research, memory):

git clone https://github.com/cortex-works/cortex-scout.git

cd cortex-scout cargo build --release --manifest-path mcp-server/Cargo.toml --bin cortex-scout-mcp

This works from the repository root because the manifest path is explicit.

Full build (includes hitl_web_fetch / visible-browser HITL):

cargo build --release --manifest-path mcp-server/Cargo.toml --all-features --bin cortex-scout-mcp

If you also want the optional HTTP server binary, build it explicitly with cargo build --release --bin cortex-scout.

Local MCP smoke test:

python3 publish/ci/smoke_mcp.py

This runs a newline-delimited JSON-RPC stdio session against the local cortex-scout-mcp binary and exercises the main public tools with safe example inputs.

---

MCP Integration (VS Code / Cursor / Claude Desktop)

Add a server entry to your MCP config.

VS Code (mcp.json — global, or settings.json under mcp.servers):

The hard timeout guard vars below are required in MCP configs. They are the safety rail that prevents a bad page, stalled browser launch, or stuck scrape stage from holding the whole MCP session open indefinitely.

// mcp.json (global): top-level key is "servers"

// settings.json (workspace): use "mcp.servers" instead { "servers": { "cortex-scout": { "type": "stdio", "command": "env", "args": [ "RUST_LOG=warn", "CORTEX_SCOUT_TOOL_TIMEOUT_SECS=90", "CORTEX_SCOUT_TOOL_TIMEOUT_SECS_SCRAPE_URL=90", "CORTEX_SCOUT_TOOL_TIMEOUT_SECS_SEARCH_STRUCTURED=120", "CORTEX_SCOUT_TOOL_TIMEOUT_SECS_VISUAL_SCOUT=45", "CORTEX_SCOUT_BROWSER_LAUNCH_TIMEOUT_SECS=12", "CORTEX_SCOUT_BROWSER_TAB_PROBE_TIMEOUT_SECS=4", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS=20", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_CDP_INITIAL_ATTEMPT=25", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_CDP_RETRY_ATTEMPT=25", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_FORCED_CDP_ATTEMPT=25", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_NATIVE_CDP_FALLBACK=25", "SEARCH_ENGINES=google,bing,duckduckgo,brave", "LANCEDB_URI=/YOUR_PATH/cortex-scout/lancedb", "HTTP_TIMEOUT_SECS=30", "MAX_CONTENT_CHARS=10000", "/YOUR_PATH/cortex-scout/mcp-server/target/release/cortex-scout-mcp" ] } } }

Default behavior is direct/no-proxy. Add IP_LIST_PATH and PROXY_SOURCE_PATH only if you want proxy tools available. If you want proxy_control available without routing normal traffic through proxies, point IP_LIST_PATH at an empty ip.txt file and let agents populate it on demand.

Important: Always use RUST_LOG=warn, not info. At info level, the server emits hundreds of log lines per request to stderr, which can confuse MCP clients that monitor stderr.

Windows: Windows has no env command. Use the command+env object format instead — see docs/IDE_SETUP.md.

With deep research (LLM synthesis via OpenRouter / any OpenAI-compatible API):

{

"servers": { "cortex-scout": { "type": "stdio", "command": "env", "args": [ "RUST_LOG=warn", "CORTEX_SCOUT_TOOL_TIMEOUT_SECS=90", "CORTEX_SCOUT_TOOL_TIMEOUT_SECS_SCRAPE_URL=90", "CORTEX_SCOUT_TOOL_TIMEOUT_SECS_SEARCH_STRUCTURED=120", "CORTEX_SCOUT_TOOL_TIMEOUT_SECS_VISUAL_SCOUT=45", "CORTEX_SCOUT_BROWSER_LAUNCH_TIMEOUT_SECS=12", "CORTEX_SCOUT_BROWSER_TAB_PROBE_TIMEOUT_SECS=4", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS=20", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_CDP_INITIAL_ATTEMPT=25", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_CDP_RETRY_ATTEMPT=25", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_FORCED_CDP_ATTEMPT=25", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_NATIVE_CDP_FALLBACK=25", "SEARCH_ENGINES=google,bing,duckduckgo,brave", "LANCEDB_URI=/YOUR_PATH/cortex-scout/lancedb", "HTTP_TIMEOUT_SECS=30", "MAX_CONTENT_CHARS=10000", "OPENAI_BASE_URL=https://openrouter.ai/api/v1", "OPENAI_API_KEY=sk-or-v1-...", "DEEP_RESEARCH_LLM_MODEL=moonshotai/kimi-k2.5", "DEEP_RESEARCH_ENABLED=1", "DEEP_RESEARCH_SYNTHESIS=1", "DEEP_RESEARCH_SYNTHESIS_MAX_TOKENS=4096", "/YOUR_PATH/cortex-scout/mcp-server/target/release/cortex-scout-mcp" ] } } }

Multi-IDE guide: docs/IDE_SETUP.md

---

Configuration (cortex-scout.json)

Create cortex-scout.json in the same directory as the binary (or repository ro