Case study · DocVal

How I measure accuracy in document-extraction AI pipelines

Parvez Ahmed · Numbers current as of the held-out re-run of 18 July 2026, after the per-page extraction fix.

DocVal's extraction took a day or two. Knowing when it was wrong took the rest. This is how I measure it, and what measuring taught me.

Three layers, in order of trust

1. Schema, cheapest. Every extraction lands in one Pydantic model: typed dates, decimal amounts, one of debit or credit per row.
2. Deterministic validation, free on every document. Balances must reconcile, the running balance must chain, dates must stay in the period. No model in the loop, so no false authority.
3. The eval harness, costs money, run on purpose: ground truth on a held-out set.

Decisions that mattered

Matches are earned, not fuzzy. A predicted row matches gold only on exact date and exact signed amount; description similarity only breaks ties. In finance a wrong amount is the error.

Split by source. The held-out F1 is 0.63. The split is the real story:

Beware circular evals. My synthetic generator and my native parser know the same layouts, so scoring one on the other partly measures agreement between two things I wrote. Only real third-party documents get full trust.

Twin leakage. The public dataset ships each statement twice, as a PDF and as a scan with identical ground truth. A random split leaks one twin into held-out. Twins now always share a split.

Cost is a metric. Every eval row records tokens and dollars: $0.008 per document, retries billed.

The regression gate

CI runs a free, offline subset on every push against a committed baseline, with a 1-point tolerance:

uv run python -m eval.run_eval --manifest tests/fixtures/ci_manifest.jsonl --no-vision
uv run python -m eval.compare results.json eval/baseline_ci.json

Nobody runs evals by hand. A red ✗ on a PR survives a deadline.

What the harness caught

• The output cap. 10 of 13 held-out errors were the model hitting a 32k output-token cap on dense documents. Per-page extraction took real-scan F1 from 0.12 to 0.42 and errors from 13% to 1%, at 1.6× the cost.
• Temperature 0 isn't deterministic. The same document flipped pass→fail between identical reruns. Run an eval once and some of your improvements are noise.
• Flaky providers look like bad models. Separating aborted streams (retry) from real model failures (fix) took the error rate from 36% to 13% without touching the model.
• A benchmark with the wrong answer key. A public bank-statement benchmark scored my submissions near zero. To prove the key was wrong I hand-transcribed one statement exactly: it scored 0.146, with 0/12 amounts right on a perfect 12/12 row alignment. (Issue filed.) They regenerated the ground truth, and the pipeline now scores 0.906 on it. Verify the third party too.

The numbers

What I learned

1. Deterministic checks beat an LLM judge. Arithmetic doesn't hallucinate.
2. Split accuracy by source; averages hide exactly what you need to know.
3. Separate transient failures from persistent ones before fixing anything.
4. Your worst honest number is your roadmap.