Quellgeist Commits

Quellgeist

First-line incident triage you can trust: ranked root-cause hypotheses where every claim cites a real evidence handle — and the agent abstains rather than guess.

Quellgeist is a model-agnostic AI agent for first-line production-incident triage. It runs a legible JSON-action ReAct loop over read-only tools (structured logs + recent deploys + metric time-series), then emits a structured Diagnosis: confidence-ranked root-cause hypotheses, each backed by a structured evidence handle (LogRef.id / CommitRef.sha / MetricRef.id) the agent actually saw — never free text. Two ideas set it apart:

• Cite-by-structured-handle. Evidence is a checkable handle, not a sentence, so a fabricated citation is *measurable* and deterministically rejected by a keyless fabrication check — not a matter of fuzzy string-matching.
• Abstain-over-hallucinate. A confidently-stated wrong cause is the worst possible answer, so *"insufficient evidence"* is a first-class outcome.

Status: Wave 4 complete — the fine-tune works. The DR-0020 QLoRA fine-tune

of the local reasoner (Qwen3-4B, served via Ollama) took it from the base's

0/16 holdout to 12/16 — zero fabrication, zero speculative-filtering, and

*cheaper* than the base — while beating a 31B frontier (Gemma-4-31B, 10/16)

on the same holdout at $0, fully offline. Non-memorisation is triangulated

three ways (fixtures ≈ holdout; core-fresh ≥ core-overlap; structure probe 7/10).

Two honest limits: the resource_exhaustion class didn't transfer (0/N; the

frontier passes it), and adversarial-abstention recall is 6/12 at the system

level — a ceiling the 31B frontier *shares* (also 6/12), not a fine-tune

regression. When this agent misses it's *incomplete* or *too cautious*, never

confidently fabricating. See

Status & roadmap · fine-tune case study.

Why it's different

What it is / what it's NOT

• It is: a first-line *triage* agent — ranked, evidence-cited root-cause hypotheses (or an honest abstention) from read-only logs/deploys/metrics, over a model-agnostic loop that runs on a hosted frontier model or a local 4B.
• It is NOT: an autonomous remediator (it never mutates prod — resolution verification is a deferred, cut-first wave); a production-hardened service (the demo is a deliberate toy); or a general-purpose agent. The holdout it's measured on is *out-of-vocabulary but in-structure* — not a claim about unseen incident shapes or real production data.

Quickstart (~30 seconds to a broken service + structured logs)

Requires uv and Python 3.12+.

See a real-shaped diagnosis in one keyless command (no model, no API key):

uv sync && uv run quellgeist diagnose --demo     # renders the demo incident's cited postmortem

Then run the full loop against the live toy service:

uv run uvicorn demo.app.main:app          # 1. start the toy service (leave running)

--- in a second shell, from the repo root ---

uv run python -m demo.chaos.bad_deploy # 2. inject a simulated bad deploy curl -s localhost:8000/login # 3. trip /login -> 500s + structured error logs uv run quellgeist diagnose --show-trace # 4. diagnose live (needs a model; see below)

uv run python -m demo.chaos.reset # back to a green slate

The live step needs a reasoner — see Running the model. Without a key, quellgeist diagnose exits 1 with a one-line error + hint (never a traceback); --demo always works keyless and renders the same output shape deterministically from gold.

Architecture

A custom, legible loop is the orchestration layer; the three read-only tools are the evidence interface; the Diagnosis schema is the contract that the postmortem renderer and the eval judge both read.

flowchart TD

trigger(["incident trigger - CLI"]) --> loop model["reasoner via LiteLLM (Gemini or local Qwen, swappable)"] -. "chat completion" .-> loop

subgraph loopbox["model-agnostic JSON-action ReAct loop"] loop["run_loop() decide, call tool, observe, repeat"] end

loop -- "query_logs" --> logs["logs tool structured JSONL, stable ids"] loop -- "get_recent_commits" --> commits["commits tool deploy_log.json, shas"] loop -- "query_metrics" --> metrics["metrics tool time-series, named series"] logs -- "rows + ids" --> loop commits -- "commits + shas" --> loop metrics -- "series + names" --> loop

loop --> diag["Diagnosis (schema.py) ranked hypotheses citing LogRef.id / CommitRef.sha / MetricRef.id, or abstains"] diag --> pm["postmortem renderer deterministic Markdown"] diag --> judge["eval judge fixture scenarios, CI gate"]

All three tools are also exposed as MCP servers over stdio (python -m quellgeist.servers.logs_mcp, …commits_mcp, …metrics_mcp). The agent currently reuses the same tool *functions* in-process behind a ToolSpec registry; a stdio MCP-*client* path (the agent driving the servers over the wire) is on the roadmap (DR-0010).

Deep dive: docs/architecture.md walks the full

pipeline (loop → tools → verifier → postmortem), a sequence diagram, the module

map, and the cross-cutting design decisions.

The servers publish to the Official MCP Registry on each tagged release (see docs/publishing.md); once published each is runnable with uvx --from quellgeist quellgeist-logs-mcp (or …-commits-mcp / …-metrics-mcp).

Example session

Inject the bad deploy — it drops a marker that flips verify_token into a NoneType regression and writes a deploy_log.json whose offending commit landed just before the errors (illustrative stdout — the timestamp reflects when you run it; paths shown relative to the repo root):

$ uv run python -m demo.chaos.bad_deploy

injected bad deploy a1b2c3d (touched demo/app/auth.py) at 2026-06-24T12:22:43Z marker: demo/.bad_deploy deploy log: demo/deploy_log.json next: hit /login to generate the 500s, then quellgeist diagnose

With a reasoner configured, quellgeist diagnose reads the logs + deploys and emits a postmortem. The CI environment has no validated model key (DR-0012), so the diagnosis below is rendered from gold — built deterministically from the fixture's labelled cause and evidence handles via render_postmortem, *not* live model output:

# Incident Postmortem (rendered from gold)

Root-cause hypotheses

1. Bad deploy a1b2c3d (10:01:50Z) refactored auth.py and introduced a NoneType error in verify_token; /login 500s begin ~20s later at 10:02:12Z. (confidence: 1.00)

Evidence:

• log #2
• commit a1b2c3d

Reproduce that render yourself (no model needed):

uv run python - <<'PY'

from evals.scenarios.generator import load_scenario from quellgeist.agent.schema import Diagnosis, Hypothesis from quellgeist.output.postmortem import render_postmortem

s = load_scenario("evals/scenarios/fixtures/bad_deploy_0001.json") gold = Diagnosis(hypotheses=[ Hypothesis(cause=s.gold_cause, confidence=1.0, evidence=s.gold_evidence_refs) ]) print(render_postmortem(gold, title="Incident Postmortem (rendered from gold)")) PY

The point isn't the prose — it's that `log #2` and `commit a1b2c3d` are exact handles into the real signals, not paraphrases. A live run additionally fills in a one-line summary and suggested actions, and abstains outright when the evidence is too weak to name a confident cause.

Write the postmortem to a file with --out postmortem.md, or as a self-contained HTML page with --out postmortem.html (or --format html) — same deterministic render, no external assets.

Running the model

The reasoner is any LiteLLM model string, selected by --model or the QG_MODEL env var (default gemini/gemini-3.5-flash). Provider keys are read from the environment by LiteLLM; nothing is stored in the repo.

export QG_MODEL="gemini/gemini-3.5-flash"

export GEMINI_API_KEY="…" uv run quellgeist diagnose --show-trace

Or fully local and offline via Ollama — the intended home default (DR-0008; exact artifact pinned in DR-0019), no API key involved:

ollama pull qwen3:4b-instruct-2507-q4_K_M

export QG_MODEL="ollama_chat/qwen3:4b-instruct-2507-q4_K_M" uv run quellgeist diagnose --show-trace

Base vs tuned — important. The ollama pull above is the base Qwen3-4B:

the honest safe *floor* — it scores **0/16 on the holdout and abstains on

everything, never fabricating (DR-0019). The 12/16** headline is the

DR-0020 fine-tune (quellgeist-qwen3-dr0020), which you build + serve via

finetune/README.md (a free-Colab QLoRA run →

ollama create). Until that tuned GGUF is published for a one-line pull, the

base model is what a plain ollama pull gives you — safe, not yet useful. Use a

hosted model (above) or the fine-tune to see live diagnoses.

Heads-up (DR-0012): a Gemini key on an unvalidated, no-billing project returns 429 limit: 0 on current models, so the shipped CI gate is deliberately keyless and model-driven evals are key-gated and run out-of-band (DR-0015). At home the intended default reasoner is a local Qwen3-4B via Ollama (DR-0008).

Running the eval (reasoner + verifier + LLM-judge)

The fixture eval scores the reasoner with a deterministic keyword judge + a zero-fabrication check (the keyless gate), and can additionally run two model layers (DR-0016): a verifier that confirms cited evidence supports each hypothesis (forcing abstention otherwise) and an advisory LLM-judge rubric.

export GEMINI_API_KEY="…"

export QG_MODEL="gemini/gemini-3.5-flash" QG_VERIFY=1 QG_JUDGE_LLM=1 \ QG_MIN_CALL_INTERVAL_S=6 \ # pace calls under the free-tier RPM (avoids 429 bursts) uv run python -m evals.run_evals

`QG_VER