Why your contract RAG returns garbage — and a reproducible fix

TL;DR: Dirty contract PDFs — mojibake, hyphenation, repeated headers, near-duplicates — quietly wreck RAG. On a committed synthetic set of 8 contracts, cleaning raises the data-quality score from 0.69 to 0.94 (+0.25) and extraction field-F1 from 0.14 to 0.98 (+0.85) — and you can reproduce every number with one command, no API key.

What "garbage in" looks like

Real enterprise documents arrive with utf-8/latin-1 mojibake, words split across line breaks, page headers repeated on every page, and near-duplicate blocks. Retrieval returns noise; extraction cites the wrong block.

Data-quality score, dirty vs cleaned

The fix: clean to a typed Document IR, then extract with attribution

Every parser converges to one Document IR. An ordered, pure clean pipeline fixes unicode, de-hyphenates, strips repeated headers, de-dupes, and normalizes whitespace. An explainable quality score flags what still needs review. Extraction then returns sourced, confidence-scored facts — every value a verbatim span of its cited block.

Extraction field-F1, dirty vs cleaned

Reproduce it yourself

git clone <repo> && cd contract-rag
uv sync --extra dev --extra benchmark
python -m contract_rag.benchmark

You will get the same field-F1 (0.14 → 0.98) and quality (0.69 → 0.94) numbers shown above.

Honest caveat

The dirt here is simulated by a seeded corruption suite on synthetic contracts, so this proves the pipeline's recovery behavior end-to-end — not real-world OCR accuracy. On real labeled contracts (CUAD, download-gated) the same cleaning lifts field-F1 from 0.33 to 0.70.

Get a free diagnosis

If your RAG returns garbage on real contracts, we will run this before/after on one of your de-identified documents and show you the numbers. Open an issue or get in touch.