IPCC AR6 Climate Retrieval

Climate-Risk Analyst Agent

Ask a plain-language question about heat, extreme rain or wind anywhere on Earth, and get back a typed, cited risk report built from live forecast data, 60+ years of ERA5 extremes and IPCC AR6. When a question falls outside what it can check, it refuses instead of guessing.

Live demo: https://climate-risk-agent-714882950125.us-central1.run.app/ (a public Google Cloud Run deployment).

What it does

• Answers for any location: type a place name to geocode it, or enter a latitude and longitude directly.
• Puts a live forecast next to 60+ years of ERA5 annual maxima at that point, and cites the IPCC AR6 pages it used by number.
• Bands the risk on that location's own return-level curve instead of a fixed threshold.
• Reports a confidence that says what it is made of: measured forecast skill at that lead day, the climatology fit, and whether a citation backed the answer.
• Quotes the AR6 Chapter 12 projected change for the reference region containing the point, verbatim.

The measurement moat

• ERA5 GEV hazard statistics. Every hazard number comes from a Generalized Extreme Value distribution fitted to 60+ years of ERA5 annual maxima at the query location. Every return level ships with a 90% bootstrap confidence interval.
• Risk levels are local rarity, not a fixed threshold. The band is the return period of the forecast peak on that location's own curve: below the 2-year level is low, 2 to 10 years moderate, 10 to 50 high, at or above 50 severe. The same 46 °C peak can band low where an ordinary year already reaches it and severe where it is unprecedented, off the same code. Each report carries the sentence the band was decided on, naming the bracketing levels and the estimated return period. A location with no fitted climatology falls back to absolute cutoffs and says so rather than passing them off as local.
• Non-stationary GEV. Alongside the stationary fit, a drifting-location GEV checks whether the climate at that location is actually warming, using a likelihood-ratio test to decide. When the trend is real, return levels are reported "effective" at the latest year rather than averaged across six decades. Berlin comes back at about +0.75 °C per decade (p < 0.0001) with effective levels. Delhi comes back stationary, with no significant trend at either Delhi point the app uses (p = 0.46 at the UI's preset Delhi point, p = 0.84 at the geocoded city point about 5 km away), which agrees with the published literature on aerosol masking suppressing South Asian heat trends. The slopes, p-values, exact coordinates and code version are in evals/results/warming-trend-2026-10-21.json, written by scripts/measure_warming_trend.py from a fresh ERA5 1960 to 2022 fit.
• IPCC AR6 RAG with citations that have to hold up. Every citation is checked structurally against the pages actually retrieved for that question. If the model cannot point to a real retrieved page, it refuses rather than cite one anyway.
• Projected change, quoted rather than generated. Each report carries the AR6 WG1 Chapter 12 climatic impact-driver projection for the reference region containing the point: one verbatim Ch.12 sentence, its direction, the IPCC's own calibrated confidence phrase, and the warming level or period it names. All of it is extracted by regex from retrieved chunks, with no model in the loop. When the chapter assesses no such change for that region and hazard, the section is absent and the report says regional projections were not found in the corpus.

The same report, scrolled down: the ERA5 return-level table with bootstrap confidence intervals, the warming-trend banner, and the measured cost and latency for that run.

How it works

A free-text question moves through parsing, geocoding and AR6 region mapping before it reaches the agent. From there a five-node LangGraph agent (plan, call, research, project, synthesize) either produces a typed RiskReport or refuses.

flowchart LR

Q[free-text question] --> PARSE[parse location · hazard · horizon] PARSE --> GEO[geocode Open-Meteo] GEO --> REGION[map to IPCC AR6 region reference regions v4] REGION --> PLAN[agent: plan scope check] PLAN -->|unsupported hazard| REFUSE[refusal valid typed output] PLAN --> CALL[agent: call forecast] CALL --> RESEARCH[agent: research] RESEARCH --> PROJECT[agent: project AR6 Ch.12 CID projection] PROJECT --> SYNTH[agent: synthesize] SYNTH --> REPORT[RiskReport typed · cited · grounded]

subgraph HAZARD [ERA5 to GEV hazard statistics] ERA5[ERA5 60+ yr annual maxima] --> GEV[stationary + drifting-location GEV] GEV --> LRT[likelihood-ratio test is the trend real?] LRT --> LEVELS[return levels + 90% bootstrap CI effective at latest year if trend holds] end CALL -.-> HAZARD HAZARD -.-> SYNTH

subgraph RAG [IPCC AR6 RAG] HYBRID[BM25 + dense hybrid RRF fusion] --> VALID[citations validated against retrieved pages] end RESEARCH -.-> RAG RAG -.-> SYNTH

The UI shows the same run as seven named steps, in this order (agent/progress.py): resolving location, fitting 60 years of ERA5 extremes, checking the question is in scope, fetching the forecast, searching IPCC AR6, reading Chapter 12 projections for your region, writing the cited report.

Evaluation

• Dev set: 60 questions, used to steer development choices like chunking and retrieval configuration. The 15 cid-table questions arrived with the Chapter 12 projection layer, and the set was re-frozen under a new SHA-256.
• Test set: 105 new questions, written after the dev set existed, never used to tune anything, frozen by SHA-256 so neither set can quietly change. 70 are answerable (24 single-page, 22 regional table, 15 multi-page, 9 duplicate region) and 35 are built to be refused (17 premise injection, 10 out-of-scope hazard, 8 out-of-corpus).

Dev-set retrieval, all 60 questions (49 answerable), measured 2026-09-02:

Per slice, hybrid R@3: single-page 92% (n=12), regional table 90% (n=10), multi-page 80% (n=10), premise injection 75% (n=4), cid-table 67% (n=15). cid-table is the weakest slice and is why the headline is 82% across all 60 questions where it was 91% across the original 45.

Dev-set end-to-end, all 60 questions (gemini-2.5-flash, top_k 8, scope guard stage 2 off, measured 2026-09-02):

Citation validity 41/48 (85%). Numeric provenance 42/48 (88%). Grounded refusals 3/4. Cost $0.20 for the run, about $0.0034 per question, p50 latency 5.5 s across 55 model calls with 6 retried rate limits and no failures.

Held-out results (second exposure, on the exact configuration deployed):

• Retrieval, hybrid, on the 70 answerable questions (measured 2026-09-07): R@3 87%, R@5 91%, R@10 96%, that is 61, 64 and 67 of 70.
• End-to-end, all 105 questions (gemini-2.5-flash, top_k 8, measured 2026-07-22): 49 correct answers, 35 correct refusals, 21 false refusals, 0 false answers. All 35 questions built to be refused were refused; the price is 21 of the 70 answerable questions refused as well.
• Citation validity 47/49 correct answers (96%). Numeric provenance 43/49 (88%).
• The IPCC answer step cost $0.29 for the run, about $0.003 per question, p50 generation latency 3.9 s.

Reranking and query rewriting: measured, kept off. Both are wired as HybridRetriever keyword arguments and both default to off. Measured on the 45-question dev set (34 answerable), before the cid-table questions landed:

No arm beat the baseline at any k, and the cheapest of them costs about 100 times the latency. The seams stay in the code because the measurement is what makes leaving them off defensible.

Semantic scope guard, stage 2: measured, kept off. Stage 1 is lexical and runs before any model call. Stage 2 (rag/scope_semantic.py, CRG_SCOPE_STAGE2=off|embed|llm) reads only the questions stage 1 leaves undecided. Measured on the 45-question dev set, as correct answer / correct refusal / false refusal / false answer:

Zero false answers in every arm, and no arm gained anything, so the default stays off. The embed arm's one false refusal carries lexical hazard vocabulary and never reached stage 2. embed is the arm to promote, once a held-out run confirms the matrix holds. The full reasoning is in LIMITATIONS.md.

Every eval artifact records the model that produced it, because a model swap is invisible to a test suite. When the answering model was changed without re-running these evals, the benchmark caught a usefulness regression that the unit tests did not: see adr/0001-answering-model-selection.md.

Refusals are scored on a 4-cell confusion matrix (correct answer, correct refusal, false refusal, false answer). One false answer on that matrix blocks release.

The gate: it is an exit code, not an eyeball. The build fails on any false answer, on headline R@3 falling more than 3 points below the last committed run, or on the hybrid retriever being missing from the artifact, which would mean the run silently measured BM25-only. See DEPLOY.md.

Forecast skill

Every hazard number rests on a forecast, and a peak predicted six days out is not the same claim as one predicted tomorrow. That gap is measured rather than assumed. scripts/measure_forecast_skill.py reads Open-Meteo's Previous Model Runs API, which archives what the model had predicted for each hour 1 to 7 days before it happened, and scores every lead time against the model's own day-0 run over 2024-01-0