Typecast

Typecast

LLM agent that turns messy job descriptions into schema-validated, typed JSON — and measures how often that fails.

Key findings (two Qwen models via OpenRouter, 30 real job descriptions):

• Schema mode makes output valid, not correct. In setup C both models were valid 98.9% of the time, at lower cost than retry, yet end-to-end accuracy was 67.1% (7B) and 84.7% (235B).
• Error-feedback retry fixed the small model, not the large one. Setup B was 98.9% valid for the 7B but 88.9% for the 235B, which failed on jd_002 in all four setup-B runs. A count-aware retry message did not change that.
• No run reached my 90% field-accuracy goal (best: 87.3%), and runs vary. The same setup moved by about 8 points end to end between runs, so a gap between two single runs is not conclusive.

60-second walkthrough:

https://github.com/user-attachments/assets/c4a2572c-88a2-49a3-8e20-176010877154

CLI and package name: soa · MCP server on PyPI: typecast-mcp

Results

Two open-weight models through OpenRouter, 30 real job descriptions, four setups. Field accuracy is scored against 20 hand-verified gold labels (see Limitations for how they were made). With 3 runs, the figure is the mean and the brackets are min–max across the runs; with 1 run it is that single run. A = plain prompt, no retry; B = plain prompt + error-feedback retry; C = provider schema mode, no retry; D = schema mode + retry.

Full table: first-try valid, avg calls and field accuracy too

How to read it:

• "Mean of 3 runs (min–max)" (B and C) is the mean of three runs on the same 30 documents, with the lowest and highest run in brackets. It is a spread across runs, not an interval estimate. The B runs are the three runs made after the retry-feedback fix described below. The C runs are the first C run plus two repeats. "Single run" (A and D) is one draw each: the repeats show the same setup moving by about 8 points end to end between runs (235B setup B: 72.8% to 80.9%). Each run's own 95% interval (Wilson for the validity columns, bootstrap for the accuracy columns) is in the per-run table at the end of this section; the single runs of A and D have none, because they were made before the interval code existed.
• First-try / final valid: share of the 30 documents whose first reply, or any reply within 3 attempts, passed validation. Avg calls: model calls per document.
• Field accuracy (valid outputs) is measured only on documents that produced a valid posting, so it flatters a setup that fails often (235B setup A: 86.6% field accuracy, but only 70.0% of the labelled documents had a valid output to score). End-to-end accuracy counts a failed document as wrong on every field and is the number to trust when setups differ in validity.
• Cost is the provider-reported cost of one run over the 30 documents. Temperature was not set (the provider default applied, value unknown).

My targets, and whether they were met

I set three targets for setup B before running anything: final validity of 30/30, at most 1.3 calls per document on average, and at least 90% field accuracy. Result, per run (the three runs after the retry fix):

The 90% field-accuracy target was missed by every run of both models, in every setup. The best number anywhere is 87.3% (235B, setup B, first-round run).

Every run, with 95% intervals (results/summary.md, verbatim)

Rows 1-8: the first round, one run per model and setup (A, B, C, D for the 7B, then for the 235B). These were written before the interval code existed, so they have none. Rows 9-18: the repeat round, in order 7B B x3, 7B C x2, 235B B x3, 235B C x2. Intervals are in brackets. Columns: field acc is over valid outputs, coverage is the share of the 20 labelled documents with a valid output, e2e acc counts failures as wrong.

The A and D figures in the main table are rows 1-8 (A and D) above. The first-round B rows (7B: 86.7% / 96.7%, 235B: 83.3% / 86.7% first-try / final) are the runs from before the retry fix; the first-round C rows are the first of the three C runs in the main table. The aggregate over the repeats is in results/aggregate.md; the raw per-run reports are results/2026-11-04_*.json.

How it works

text ──► build prompt (schema + instructions)

│ ▼ LLM call ──► raw string │ ▼ JobPosting.model_validate_json(raw) │ ok │ ValidationError ▼ ▼ log success log failure (error + raw) return model attempt < MAX? ──► append error to messages, retry │ no ▼ raise ExtractionFailed (logged)

The schema shown to the model is generated from the Pydantic model (JobPosting.model_json_schema()), never written by hand, and the reply is checked with the same model. A failed reply goes back to the model with the validator's errors, one line per error, for up to 3 attempts. After a valid reply, two deterministic tools tidy the result (canonical skill names, a location check against the document header). They are plain code, not model calls, so they add no LLM calls and the setups below stay comparable. Details: docs/ARCHITECTURE.md.

The four setups: A plain prompt, no retry. B plain prompt, errors fed back (up to 3 attempts). C the provider's strict JSON-schema mode, no retry. D schema mode plus retry.

Why not just use native structured outputs?

Schema mode helped a lot, and it is cheaper. It also does not do the whole job.

What it fixed. Setup C was valid on the first try 98.9% (96.7%-100.0%) of the time for both models, against a single plain-prompt run (A) of 83.3% for the 7B and 70.0% for the 235B. The 30-item cap on skills was the constraint that models actually broke in plain mode; under schema mode the report counts replies that broke a constraint the schema carries, and it was 0 in every C and D run (6 C runs, 2 D runs). C also cost less than B: $0.0076 against $0.0103 for the 7B, $0.0136 against $0.0211 for the 235B (means per run).

What it does not do:

1. It guarantees shape, not meaning. The remaining failures in C were rules no JSON s