How accurate is structured extraction from real NDA contracts?

On the public Kleister-NDA benchmark (a deterministic 40-doc set of real SEC EDGAR NDAs), a deterministic rule-based extractor reaches field-F1 0.697 with source-attribution accuracy 1.0 — measured 2026-07-06 and reproducible from any Kleister checkout with one command.

Why do synthetic extraction demos overstate accuracy?

Because rules and prompts get tuned on the demo documents themselves. The same extractor that scored 0.981 on a synthetic NDA set scored 0.523 on real Kleister-NDA documents before targeted fixes. The synthetic number proves pipeline correctness; the public benchmark measures real-document extraction.

Kleister-NDA, measured: honest extraction numbers on a public benchmark

TL;DR: On the public Kleister-NDA benchmark (arXiv 2105.05796 — real SEC EDGAR NDAs, a deterministic 40-doc set), our credential-free rule extractor scores field-F1 0.523 out of the box and 0.697 after one round of targeted rule improvements, with source-attribution accuracy 1.0 in both rounds — every extracted value is a verbatim span of the block it cites. Separately, switching structured LLM output from client-side TOOLS-mode function calling to server-side schema-constrained decoding cut schema-validation failures from 12/40 (30%) to 0/40 on an identical rig. Every number in this article is a point-in-time measurement (2026-07-06), committed to the repo in content/kleister_results.toml and injected at build time — not recomputed on each site build — and reproducible with the commands at the end.

We attacked our own synthetic number first

Our NDA vertical first shipped with an author-written synthetic golden set, scoring field-F1 0.981. That number is real but proves the wrong thing: it demonstrates pipeline correctness — extraction, attribution, and metrics all work end-to-end for a new domain — not real-world accuracy, because the rules were tuned on the very documents they were scored against.

So we attacked it with a third-party public benchmark. Kleister-NDA is a set of real NDAs filed with the SEC, labeled by the benchmark's authors, not by us. The same extractor that scored 0.981 on synthetic documents scored 0.523 on real ones. That gap — 0.981 vs 0.523 — is the honest picture, and it is exactly why extraction demos on vendor-authored data should not be trusted, including ours.

Per-field results, two rounds

Kleister-NDA labels four fields. Field-F1 on labeled docs ("initial" = rules as tuned on the synthetic set; "improved" = after targeted fixes; the 40-doc set is built deterministically from train + dev-0 — the hidden test-A gold is never used):

field	labeled docs	initial F1	improved F1
party	40	0.025	0.550
effective_date	29	0.345	0.759
term	14	0.286	0.714
governing_law	40	0.925	0.925 (unchanged)

Blended field-F1: 0.523 → 0.697. Source-attribution accuracy: 1.0 in both rounds.

What the improvements actually were

Three targeted fixes, each aimed at a measured failure mode, each canonicalized on both sides (the same helper canonicalizes gold and extraction, so formatting differences never count as errors) and unit-tested:

party 0.025 → 0.550. The synthetic-set heuristic looked for explicit "Disclosing Party" / "Receiving Party" role labels — which almost never appear in real SEC NDAs. Fix: a preamble fallback that parses the "by and between …" opening clause, with a documented first-named-is-disclosing convention (explicit labels still win when present).
effective_date 0.345 → 0.759. Real filings use legalese date forms the original regex missed — "the 6th day of January, 2012", ordinals, day-month-year order, standalone letterhead dates. Fix: those forms, plus a cue-proximity finder so a date near "effective" / "entered into" beats one far away.
term 0.286 → 0.714. Word-number durations ("two (2) years") and a "shall terminate" cue.

governing_law was already at 0.925 and was left untouched. Two regression guards held throughout: the synthetic NDA eval stayed at 0.981, and the generic CUAD contract baseline is byte-identical — the NDA rules only reuse shared helpers; there is no core fork.

Source-accuracy 1.0 — by construction, not luck

Every extracted field carries a source_block_id, and the metric verifies the extracted value actually appears in the text of that block (for the multi-valued party field, every extracted entity must appear in the cited block). The rule finders only ever emit spans of the block they matched, so attribution holds by construction — accuracy stayed 1.0 even while field-F1 was 0.523. Getting the value wrong and citing it honestly is a recoverable failure mode; citing the wrong evidence is not.

Structured decoding: the bottleneck was reliability, not the model

A separate measurement, on the 40-doc CUAD contract set (not Kleister), same rig both runs — Lambda A100-40GB, Ollama, qwen2.5:32b-instruct @ 32K context:

Client-side TOOLS-mode function calling (instructor): 12/40 docs (30%) failed schema validation outright (malformed nested JSON, e.g. an extra wrapper object) and counted as misses. Field-F1 0.576, source-accuracy 0.640 — partly the model dropping the # prefix on cited block ids.
Server-side schema-constrained decoding (response_format=json_schema): 0/40 schema failures. Field-F1 0.661–0.672 (two runs; decoding is non-deterministic, treat ±0.01 as run noise), source-accuracy 0.744–0.756.

Same model, same prompts, same documents. The 30% failure class was a transport problem, and grammar-constrained decoding removes it at the transport level. The remaining source-accuracy gap is wrong_span — the right value cited from the wrong block — which no output-format constraint can fix.

Honest limitations

Kleister-NDA labels only 4 fields, and two are sparsely labeled (effective_date 29, term 14 of 40). The blended F1 averages over what is scoreable.
The remaining party misses are structural: most are person names or suffix-less gold entities the corporate-entity regex cannot emit by design.
The rule extractor is a deterministic, credential-free floor, not a capability ceiling — no LLM was used for the Kleister numbers.
The structured-decoding numbers are from CUAD contracts, not Kleister — they measure output reliability of the local-LLM path, on a different corpus.
These are point-in-time numbers (2026-07-06), committed as data and injected at build time. They are not live-recomputed like our cleaning benchmark article, because the dataset cannot be committed (below) and the GPU runs cannot execute in CI.

Reproduce it yourself

The Kleister-NDA repository states no explicit license, so its files are never committed here — point KLEISTER_DIR at your own checkout:

git clone https://github.com/applicaai/kleister-nda
git clone <repo> && cd contract-rag && uv sync --extra dev
KLEISTER_DIR=../kleister-nda uv run python -m contract_rag.verticals.nda.kleister         # build the deterministic 40-doc set
KLEISTER_DIR=../kleister-nda uv run python -m contract_rag.verticals.nda.kleister --eval  # field-F1 + source-accuracy

The set build is deterministic (train + dev-0, seeded selection), so you should reproduce field-F1 0.697 / source-accuracy 1.0 up to environment variance in the PDF parser. If your numbers differ materially, open an issue — that is the point of measuring in public.