Engineering
Teaching our AI to show its work
Closact reads dense legal and financial documents and pulls out structured data automatically. Getting that reliably right took several rounds of deliberate, measured improvement — not one silver bullet. Here's the staged approach we used.
Ivan Korostelev, CTO · July 2026
The challenge
Large language models read long, dense documents well, but they have a specific weakness: they can state a wrong answer with exactly the same confidence as a right one. There's rarely a hedge in the output that flags "this one deserves a second look."
In our own testing, we found a category of fields — the kind anchored to a specific number, date, or classification, often stated once, sometimes far from where the rest of a document's narrative sits — coming back wrong meaningfully more often than fields like names or simple yes/no flags. That gap is exactly what a legal or financial workflow can't tolerate, so it became the focus of a sustained effort rather than a one-off patch.
The approach
Seven stages, each measured before moving to the next.
No single change closed the gap. Accuracy came from stacking several independent improvements, each validated against the same test set before we trusted it.
- 1
Start from a fixed schema
Every extraction is bound to a strict, structured output shape from the first call, not a loosely-parsed block of text. This buys consistency and type safety immediately, but it doesn't buy correctness — a well-formed answer can still be the wrong answer.
- 2
Build a test set before touching the model
Before changing a single prompt, we built a large internal set of realistic documents with known correct answers, spanning different layouts and writing styles. Without that in place, “the model seems better” is a feeling, not a measurement.
- 3
Diagnose failure patterns, not just failure rates
A raw accuracy number tells you something is wrong; it doesn't tell you what. Grouping the misses revealed a handful of repeating patterns — values split across more than one mention that need to be combined, figures placed in a table or schedule far from the main narrative, and a value getting confused with a similarly-worded one nearby.
- 4
Target those patterns directly
Generic prompting improvements have a ceiling. Adding a small number of worked examples addressing the exact confusions we'd diagnosed moved the needle far more than broad instruction tuning did.
- 5
Require the model to cite its source
For the categories of fields most prone to these mistakes, the model must now quote the exact passage its answer came from — the same discipline a careful reviewer uses when highlighting a source line instead of just writing down a number.
- 6
Verify the citation automatically
A lightweight, deterministic check — no extra model call — confirms each quote actually supports the value given. It's cheap, fast, and it doesn't get tired or skip a step the way a manual spot-check might.
- 7
Retry once, automatically, on a failed check
When a citation doesn't hold up, the system asks the model to look again a single time before finalizing an answer, rather than silently shipping a guess forward.
The result
The fields most prone to these mistakes went from an open question to essentially solved.
Across the category of number, date, and classification fields this work targeted, accuracy moved from roughly 4 wrong in every 5 to essentially all of them right — and, just as important, we now have a reliable signal for the rare cases still worth a second look.
Measured against a large written test set of realistic documents built specifically to exercise these patterns. We're still working through a full-scale validation run, and results have held steady across every batch scored so far.
What's next
This round targeted the categories of fields we judged riskiest to get wrong. A handful of lower-stakes, more judgment-based fields don't have the same citation-and-check treatment yet, and still miss occasionally — they're next in line for the same seven-stage treatment.
We deliberately scoped the citation-and-verify step to the highest-risk fields first to keep it fast and practical, rather than applying it everywhere at once. Extending it further, category by category, is the main lever we have left to pull.
See how Closact closes deals faster.
Talk to our team about intake, drafting, registry, and signing on one auditable platform.