Umbra

Everyone is vibecoding. Nobody is verifying. Umbra scores it.

Umbra is a deterministic Trust Score (0–100) for AI-generated code: the vibe coding security scanner that verifies what your agent shipped, not what it claimed. One command, fully local, evidence for every finding.

→ [umbra-badge.umbrabadge.workers.dev](https://umbra-badge.umbrabadge.workers.dev) — the landing, live badges and report pages

Quickstart · Demo · The Audit · How it works · The Four Axes · FAQ · Roadmap · Contributing · Website

Why Umbra exists

Studies put exploitable vulnerabilities in 40 to 60 percent of AI-generated code, and coding agents routinely claim "all tests pass" when three do. The tooling for *writing* code with AI is a year ahead of the tooling for *trusting* it. Umbra closes that gap: SAST rebuilt for how software gets written now, plus sandboxed verification that catches what static rules cannot.

One command scans any repo an agent produced (Claude Code, Cursor, Copilot, Windsurf, Lovable) and returns a score with file:line evidence for every finding. With --deep it goes further: Umbra builds and boots the repo in a locked-down Docker sandbox, then replays the agent's own claims against reality. If the agent is lying about tests, the score is capped below passing, with receipts.

The audit: 61 vibe-coded repos, scanned

We ran Umbra over 61 public, actively-maintained AI-built repos and published everything. The Vibe-Coding Security Audit:

Mean trust score: 74/100. One in five repos fails outright. The full report has per-class deep dives with representative snippets and fixes, the complete per-repo table, and an honest methodology section — including the false positives we found in our own rules while running it, and fixed (rubric v4).

Quickstart

npx umbra-scan            # check — scans the directory you're standing in

npx umbra-scan --fix # heal — applies provably-safe fixes, shows the score climbing npx umbra-scan --setup # protect — pre-commit gate, PR checks, agent guardrails

That's the whole interface. Three verbs: check, heal, protect.

Using an AI coding agent? Umbra is built to be driven by agents, not just run by humans:

• Any agent — it reads this repo's AGENTS.md / llms.txt and knows what to do. Or tell yours: "check this repo with umbra."
• Claude Code / Kimi Code — --setup installs PreToolUse hooks so every file the agent writes is guarded before it lands.
• Claude Code, Cursor, Copilot, Windsurf — the trust-review skill makes the agent scan its own work before declaring done.
• MCP-native agents — add umbra-mcp (npx --yes -p @elberacasa/umbra umbra-mcp) and the agent gets scan_repo, guard_content, and get_score as tools.

Real output, scanning a typical vibe-coded Next.js app (fixtures/bad-app in this repo, Trust Score 30/100):

$ npx umbra-scan ./fixtures/bad-app

UMBRA TRUST SCORE: 30/100 🔴

SAFE 🔴 5/100 — 14 findings CLEAN ✅ 87/100 — 10 findings RUNS — not measured — run with --deep HONEST — not measured — run with --deep

Score computed over measured axes only (full rubric: SAFE 35%, RUNS 25%, HONEST 25%, CLEAN 15%). Rubric v4. …plus 7 further findings beyond the per-rule cap (see report)

Top findings: [safe/hardcoded-secrets] Hardcoded Supabase service_role JWT — bypasses all row level security — .env:2 [safe/hardcoded-secrets] Hardcoded Supabase service_role JWT — bypasses all row level security — lib/supabase.ts:5 [safe/supabase-antipatterns] Supabase service_role key reachable from client-side code — full database bypass for anyone who opens the bundle — .env:2 [safe/supabase-antipatterns] Supabase service_role key reachable from client-side code — full database bypass for anyone who opens the bundle — app/components/UserList.tsx:10 [safe/hardcoded-secrets] Committed environment file with secret values: .env — .env:1

Notes (low confidence — not scored): [safe/missing-rate-limit] Auth endpoint with no rate-limiting signal in the repo — brute-force / credential-stuffing exposure (heuristic) — app/api/login/route.ts:3

Badge:

The exit code is 1 when the score is below 50, so CI can gate on it.

All commands and flags (the expert layer — most users never need these)

umbra [path]               # path defaults to the current directory

umbra [path] --json # machine-readable output umbra [path] --offline # skip npm registry checks, fully local umbra [path] --deep # verify RUNS and HONEST in a Docker sandbox umbra [path] --report # write UMBRA.md: an agent-actionable task list umbra [path] --fix # apply provably-safe fixes and re-scan (score before → after) umbra [path] --dry-run # preview --fix without writing anything umbra [path] --baseline-write # write .umbra-baseline.json: grandfather current findings, gate only on new ones umbra [path] --baseline # use an explicit baseline file ("write" is shorthand for --baseline-write) umbra [path] --publish # self-report the score to the hosted badge service (live README badge) umbra setup # install everything (hooks + Action + agent guards) umbra init # only the pre-commit hook + GitHub Action umbra protect # only the agent PreToolUse hooks (--remove uninstalls) umbra guard --stdin # hook entrypoint (agents call this, not humans) umbra mcp # run the MCP server (bin: umbra-mcp)

The canonical package is @elberacasa/umbra; umbra-scan is the short alias. Same engine either way.

How it works

repo in

│ ▼ Layer 0 · static rules (17 SAFE + CLEAN rules, 0 tokens, <1s) ▼ Layer 1 · evidence gating (confidence-scored, low never moves the score) ▼ Layer 2 · --deep sandbox (Docker: build, boot, HTTP probe, claim replay) │ ▼ deterministic Trust Score + verdict + badge

Every finding carries a confidence level and file:line evidence. Only high and medium confidence findings move the score; hunches go to a notes section. The rubric is versioned (currently v3), so the same repo always gets the same score. Full math in RUBRIC.md.

The immune layer: guard the write, not just the repo

Scanning finds problems after they land. The immune layer checks every file your agent writes before it lands. umbra protect installs PreToolUse hooks into Claude Code and Kimi Code (auto-detected, one command); the same engine backs the umbra-mcp server for MCP-native agents.

flowchart LR

CC[Claude Code hook] --> E KC[Kimi Code hook] --> E MCP["umbra-mcp: guard_content"] --> E E{"guardContent(file, content) file rules + path guard"} -->|allow / warn| W[write lands] E -->|"block (exit 2)"| B["reason fed back: agent fixes the root cause"]

npx umbra-scan protect   # install the hooks; --remove uninstalls cleanly

A leaked Stripe key or an alg: none JWT never reaches the file. The path guard hard-blocks agent writes into .git/hooks and .git/config (CVE-2026-26268, the agent-planted git hook escape), and live credentials going into .env. Blocking is reserved for high-confidence critical/high findings; everything else warns, and every failure fails open. Verdicts land in ~0.2 ms, so the guard never slows the agent down. Full story: docs/immune-layer.md.

--deep: verify AI code, don't trust it

The fast scan is static. --deep is LLM code verification with evidence. Umbra copies the repo into a throwaway Docker container (no network at runtime, 512 MB / 1 CPU hard limits, 120-second kill switch), builds it, boots it, HTTP-probes its endpoints, and replays every claim found in READMEs and agent artifacts against what actually happens. Slower (minutes, not seconds) and needs a running Docker daemon. Without Docker the sandboxed axes are skipped and left out of the score; unverifiable is never punished.

Real output, deep-scanning a repo whose README lies (fixtures/claims-app, capped at 49/100 by the liar cap):

$ npx @elberacasa/umbra ./fixtures/claims-app --deep

UMBRA TRUST SCORE: 49/100 🔴

SAFE ✅ 100/100 — 0 findings CLEAN ✅ 100/100 — 2 findings RUNS — not measured — No detectable run path (no Dockerfile, no package.json start script or main entry) HONEST ⚠️ 50/100 — 2 claims failed, 2 verified, 1 unverifiable

Score computed over measured axes only (full rubric: SAFE 35%, RUNS 25%, HONEST 25%, CLEAN 15%). Rubric v4. Score capped below passing: a documented claim was verified false. Trust is the product.

Claim receipts: CLAIM FAILED: "14 tests pass" — README.md:7 — actually 3 tests pass, 0 fail CLAIM FAILED: "build passes" — README.md:9 — actually build exits 1 CLAIM VERIFIED: "All tests pass" — CLAUDE.md:3 — 3 tests pass CLAIM VERIFIED: "All tests are passing" — README.md:8 — 3 tests pass

An