contract-rag

Your contract RAG returns garbage. Here is the fix, measured.

contract-rag parses, cleans, and scores dirty contract PDFs, then extracts sourced, confidence-tiered facts you can cite by block. Every number on this page is reproducible from the public repo -- simulated inputs are labeled as such.

dirty→clean field-F1 0.333 → 0.701 quality score 0.64 → 0.97

40 real CUAD contracts, simulated dirt (seeded corruption suite)

Send us your dirtiest contract — PoC report in 48h

What it does

Parse router, including real scans

Text-coverage probing routes each document (docling / VLM / paddleOCR); per-page routing catches mixed digital+scanned contracts a whole-document router misses -- measured: 98.8% of mixed EDiTh documents misroute at least one page under single-route parsing.

Explainable quality score + HITL

A weighted, explainable score (garble, table integrity, empty blocks, OCR confidence) flags what needs human review -- and we tell you where it can't see: confidence can't flag omissions, because an omitted fact produces no block at all, not a low-confidence one. We measured that gap and built a geometric coverage signal for it.

Sourced extraction, verified, CLM-ready

Every extracted value is a literal span of its cited source block -- attribution holds by construction. verify() quarantines low-confidence or unattributed fields for review, and the same facts export straight to CSV/JSON with a best-effort Salesforce/Ironclad field mapping.

Measured on public datasets

CUAD (40 real contracts)rule extractor field-F1 0.676 (95% CI [0.594, 0.746]), source-accuracy 1.000; local schema-constrained decoding (qwen2.5-32B) 0.661–0.672 with 0/40 schema failures, vs 12/40 under standard TOOLS-mode function calling. Simulated dirt lift: field-F1 0.333 → 0.701, quality 0.64 → 0.97.
Kleister-NDA (40 real SEC EDGAR NDAs)rule extractor field-F1 0.523 → 0.697 after two rounds of targeted fixes, source-accuracy 1.0 -- the honest gap against a synthetic 0.98 golden set.
Tobacco800 (100 real scanned pages)signature-presence detector: precision 0.981, recall 0.773, F1 0.864 -- flags 33/34 unsigned documents that a naive always-signed baseline finds none of.
FinCriticalED (85 real degraded SEC pages)7.7% of 491 expert-labeled facts are absent from OCR output entirely, while the doc-level quality score still reads 0.998 -- the blind spot a confidence threshold cannot catch.
EDiTh (86 real mixed-format documents)98.8% of mixed digital+scanned documents misroute at least one page (204/464 pages) under whole-document routing -- the measurement that justified building per-page routing.

We publish negative results

  • We reproduced DAPEI-style definition injection for retrieval: zero lift through hybrid under both a hashing embedder (hybrid 0.586 → 0.586, flat) and a semantic embedder (dense-only injection cell-for-cell identical to baseline, hybrid 0.698 → 0.698); full injection actually hurt BM25 (0.690 → 0.664) and dragged semantic hybrid to 0.681. We shipped the instrumentation, not the technique.
  • We spiked an alternative local OCR engine (FrankenOCR): parity field-F1, 25–50× slower, and it hallucinates text on shredded scans. Not adopted.
  • Our own quality score has a blind spot: on FinCriticalED, 7.7% of expert facts vanish from OCR output while the score still reads 0.998. OCR confidence can't flag it either -- an omitted fact produces no block at all, not a low-confidence one. That gap is why we measure it in the open instead of hiding it.

FAQ

Why does a dirty contract PDF break RAG retrieval and extraction?

Mojibake, mid-word hyphenation, repeated page headers, and near-duplicate blocks poison chunk retrieval and confuse structured extraction -- the model retrieves noise and cites the wrong span. Cleaning to a typed Document IR before indexing or extracting fixes the root cause instead of papering over it with a bigger model.

How accurate is extraction on real contracts, not a synthetic demo?

On the public CUAD benchmark (40 real contracts) the deterministic rule extractor reaches field-F1 0.676 (95% CI [0.594, 0.746]) with source-accuracy 1.000. On real SEC EDGAR NDAs (Kleister-NDA) the same approach reaches 0.697, versus 0.981 on our own synthetic golden set -- the gap between synthetic and real-document numbers, published rather than hidden.

Does this handle scanned and mixed digital/scanned contracts?

Yes. A text-coverage probe routes each page to the right parser (native-digital vs OCR), which matters: 98.8% of real mixed digital+scanned documents misroute at least one page under whole-document routing, which is why routing is per-page, not per-document. Scanned-page signature detection is also built and measured (Tobacco800: F1 0.864).

What does 'sourced and verified' actually mean?

Every extracted value is required to be a literal span of the block it cites (source_block_id) -- attribution holds by construction, not by prompting. A verify() guard quarantines low-confidence or unattributed fields for human review instead of shipping a silently wrong answer.

Can extracted facts flow into our CLM (Salesforce, Ironclad)?

Yes -- a CSV/JSON export carries doc_id, field name, value, source_block_id, confidence, risk tier, and a verified flag, with a best-effort Salesforce/Ironclad field-name mapping for import convenience (not an official schema).

Why do you publish negative results instead of just the wins?

Because a vendor that only shows you the numbers that worked is asking you to trust marketing, not evidence. We measured DAPEI-style definition injection and it didn't help; we spiked an alternative OCR engine and didn't adopt it; we measured our own quality score's blind spot to OCR omissions. All three are on this page, next to the results that did work.