Agent Observability

agent-observability

Open source agent observability — see what your agents did, why they failed, and what it cost. Runs locally. No cloud required.

Quick Start

One command. No manual config. No copy-paste.

npx agent-obs@latest setup

This auto-detects your AI agent (opencode, Claude Code, Cursor) and configures everything:

• Adds agent-obs to your MCP config
• Adds the self-reporting instruction
• Tells you what to do next

Then restart your agent. Every action will now self-report.

agent-obs dashboard    # http://localhost:9400 — see your sessions appear

What you'll see

After running a task in your agent, the dashboard shows:

• Every tool call with duration and status
• A-F session grade (how efficient was your agent?)
• Cost per session in tokens and dollars
• Full audit trail of what was read, edited, and executed

Manual setup (if setup can't detect your agent)

opencode: Add to opencode.json:

{

"mcp": { "agent-obs": { "command": "npx", "args": ["-y", "agent-obs@latest", "server"], "enabled": true } } }

Claude Code: Create .mcp.json:

{

"mcpServers": { "agent-obs": { "command": "npx", "args": ["-y", "agent-obs@latest", "server"] } } }

Cursor: Add to Cursor Settings > MCP:

{

"mcpServers": { "agent-obs": { "command": "npx", "args": ["-y", "agent-obs@latest", "server"] } } }

See Agent-Specific Setup below for detailed per-platform instructions including self-reporting directives.

Note: All data is written to a local SQLite database in ~/.agent-observability/. No data leaves your machine.

Proxy mode (fallback)

For agents that can't self-report, wrap any MCP server and every tool invocation gets traced automatically:

agent-obs proxy --desc "fix login bug" -- npx @modelcontextprotocol/server-filesystem /tmp

Proxy mode captures only MCP tool calls (~30% of typical agent actions). Prefer the setup / MCP server approach above.

Agent-Specific Setup

Claude Code

1. MCP Config — Create .mcp.json in your project root:

{

"mcpServers": { "agent-obs": { "command": "npx", "args": ["-y", "agent-obs@latest", "server"] } } }

1. Self-reporting instruction — Create .claude/instructions.md (or reference the existing SKILL.md under .claude/skills/agent-obs/SKILL.md):

After every tool call, report to agent-obs's log_tool_call.

Start each task with start_session. End with end_session.

1. Verify — Run agent-obs dashboard, open http://localhost:9400, and look for your session after the agent completes a task.

Cursor

1. MCP Config — Add to Cursor's MCP settings (~/.cursor/mcp.json, or Settings → MCP → Add new global MCP server):

{

"mcpServers": { "agent-obs": { "command": "npx", "args": ["-y", "agent-obs@latest", "server"] } } }

1. Self-reporting instruction — Create .cursorrules (or .cursor/rules/agent-obs.md):

After every tool call, you MUST call the agent-obs MCP server's log_tool_call tool.

Include toolName, status (success/error), outputSummary, and durationMs. Start each session with start_session and end with end_session.

1. Verify — Run agent-obs dashboard, open http://localhost:9400, and look for your session after the agent completes a task.

Architecture

                   ┌──────────────────────────┐

│ agent-obs server │ │ (self-reporting) │ │ │ │ Agent calls obs_record_* │ │ tools directly via MCP │ │ • tool calls │ │ • token usage │ │ • decisions │ │ • grades │ └──────────┬───────────────┘ │ ▼ ┌──────────────────────────┐ │ agent-obs proxy │ │ (fallback — MCP-only) │ │ │ MCP Client ───────▶│ intercepts tool calls ──▶│ MCP Server (Claude/Cursor) │ logs to SQLite │ (filesystem, └──────────┬───────────────┘ github, etc.) │ ▼ ┌──────────────────────────┐ │ ~/.agent-observability │ │ SQLite DB │ │ │ │ • sessions │ │ • tool_calls │ │ • token_usage │ │ • decisions │ │ • grades │ └──────────┬───────────────┘ │ ▼ ┌──────────────────────────┐ │ agent-obs dashboard │ │ (port 9400) │ │ │ │ GET /api/sessions │ │ GET /api/sessions/:id │ │ GET /api/search?q= │ │ GET /api/stats │ │ POST /api/export │ └──────────────────────────┘

API Reference

REST Endpoints (Dashboard)

MCP Tools (Server Mode)

What Gets Tracked

Every tool call

• Tool name — e.g. read_file, execute_command, search_code
• Server — which MCP server handled it
• Duration — wall-clock milliseconds
• Status — success or error
• Input — full arguments passed to the tool
• Output — full result returned (truncated at 64KB for storage)

Token consumption

• Input tokens — prompt and context
• Output tokens — generated response
• Total tokens — sum per session
• Per-call breakdown — token delta for each LLM round-trip

Cost estimation

Costs are estimated based on published API pricing:

Costs are tracked per session and displayed in the dashboard. You can add custom model pricing via ~/.agent-observability/models.json.

Session grades

Every session receives a letter grade based on efficiency and correctness:

Decision points

When the agent has multiple possible actions, it can record why it chose one over another. Example:

{

"context": "need to read a file — it could be at src/config.ts or lib/config.ts", "options": ["read_file src/config.ts", "glob **/config.ts", "grep 'CONFIG' *.ts"], "chosen": "glob **/config.ts", "reasoning": "File might not exist at the expected path; glob guarantees finding it regardless of location" }

Decision points create an audit trail of agent reasoning, making it possible to understand not just what happened, but why.

Full audit trail

Every action is timestamped and linked to a session. The audit trail answers:

• Who (which agent/user) accessed what data?
• When did each tool call happen?
• What data was read or modified?
• Was the action authorized?

Grade System

Grades are assigned manually by the agent via obs_record_grade or automatically by the dashboard based on session statistics:

• A = Clean — No issues detected. Every tool call succeeded, token usage was efficient relative to task complexity, and the session completed its stated goal.
• B = Minor issues — Some inefficiencies (e.g., reading the same file twice, calling a tool that returned empty results and then trying it again). No failures.
• C = Inefficient — Needs attention. Redundant tool calls, excessive token consumption, or recoverable errors that were eventually resolved.
• D = Risky — Significant problems. Multiple failed tool calls, unusually high cost for the task, or the agent chose demonstrably wrong tools.
• F = Failed — Errors occurred that prevented task completion, or the agent abandoned the session.

Grades accumulate over time. The /api/stats endpoint shows your average grade and grade distribution.

Why This Exists

AI agents are a black box. You tell Claude Code or Cursor to fix a bug, refactor a module, or add a feature — and minutes later you have a diff. But what actually happened in between? How many tool calls did it make? How many tokens did it burn? Which files did it read? Did it try three approaches before landing on one? You have no idea.

This isn't just academic curiosity. Without observability:

• You can't estimate costs. A session that feels quick might have burned $5 in API calls. A session that felt slow might have cost $0.50. You're operating blind.
• You can't debug failures. The agent produced broken code — was it because it read the wrong file? Used a deprecated API? Misunderstood the task? Without traces, you're guessing.
• You can't improve your prompts. Is the agent reading too m