DocVal

I had a pile of receipts and bank statements I wanted as data, and open-source tools that claim to do it.

Then a friend told me their company offers this as a service: people check every receipt and statement by hand and type it into the client’s system. OCR still isn’t trusted with it. I wanted to know why, so I built one.

Reading the document turned out to be the easy half; a vision model does it in an afternoon. The hard half is knowing when it read it wrong, and that is why the humans are still there.

A bank statement is one of the rare documents that grades itself. Opening balance plus credits minus debits must equal closing. The running balance must hold row by row. Dates must stay in order and inside the period.

So instead of asking a second model whether the first did well, DocVal does the sum. If it doesn’t add up, the extraction ships flagged, not clean.

1. TriageDoes the PDF have a real text layer, or is it pixels?
2. ExtractNative PDFs go through a deterministic pdfplumber parser. Scans go to Gemini 2.5 Flash-Lite vision, per page.
3. SchemaBoth paths land in one Pydantic model. Money is Decimal, never float.
4. ValidateBalance continuity, running-balance chain, date sanity. Pure functions, no model in the loop.

The model returns plain strings and Python turns them into types. Models copy what is printed no matter how you ask, so $ becomes USD in code, with a test, not in a prompt.

Held-out transaction F1 is 0.63, which on its own says almost nothing. Split by source, it says everything.

The model reads clean synthetic pages almost perfectly and garbles dense real ones: shifted columns, swapped debits and credits, misread digits. The validator flags exactly those, so only 0.25 of documents pass. On hard documents that’s the system working; a high pass rate would mean weak checks.

The first run scored 0.12 on real scans. The eval said why: 10 of 13 failures were the model hitting its 32k output-token cap on dense documents and dropping rows. Not bad reading, a structural limit. Now dense documents are read a page at a time and merged, with the running balance checking the seams.

• 0.42real-scan F1, up from 0.12
• 1%error rate, down from 13%
• 1.6×the cost per document of that fix
• $0.008mean cost per document, retries billed

Cost sits next to accuracy on purpose: a gain that multiplies the bill is a different decision from a free one.

Run over the dataset itself, the validator found that about 12% of transaction rows have running balances that don’t add up, in the published ground truth and in the PDFs. A pipeline that trusted its inputs would have scored itself against wrong answers.

• Transaction F10.630.92 synthetic, 0.42 real
• Header field accuracy0.87
• Validation pass rate0.25
• Error rate1%
• Mean cost per document$0.008
• Mean latency per document70.9 s

100 held-out documents on gemini-2.5-flash-lite, re-run on 18 July 2026.

Arithmetic doesn’t hallucinate.

The people doing that data entry aren’t there to read, they’re there to vouch. A model can’t vouch for itself; a document with its own checksums can. Check with deterministic code, split every number by source, and publish the worst one: 0.42 is where the work is.

Free precomputed samples, or a live upload of your own statement, up to 5 pages and 10 MB.

How the accuracy is measured is its own story: the eval case study.