TL;DR: We measured OCR omission against expert fact-level ground truth on 85 real degraded SEC EDGAR pages (the FinCriticalED dataset). 7.7% of 491 expert-labeled facts appear nowhere in the OCR output — silently dropped — while the document-level quality score reads 0.998. Worse: no OCR confidence threshold can flag the loss, because an omitted fact doesn't produce a low-confidence block — it produces no block at all. The fix is a coverage signal, not a confidence threshold; we built a geometric one and report where it works (5.4× signal on occluded signature zones) and where it doesn't (single missing numbers). Every number here is a point-in-time measurement (2026-07-07), committed in content/fincritical_results.toml and reproducible with the commands at the end.
One number pair, from the same 85 pages:
| what the pipeline reports | what the ground truth shows |
|---|---|
| document quality score 0.998 | 7.7% of expert-labeled facts missing from the output entirely |
The quality score is near-perfect exactly when one in 13 critical facts has vanished. That is not a bug in one formula — it is a structural blind spot shared by essentially every OCR quality signal in production use: garble rate, empty-block rate, and confidence all score what the engine emitted. A fact the engine never emitted touches none of them.
We first hit this blind spot on 100 real scanned documents (Tobacco800, 1940s–90s typewritten and faxed pages), where modern OCR read even ugly scans at mean quality 0.987 with zero pages flagged for review. FinCriticalED let us put a number on what that score misses, because it carries something rare: expert annotations of every critical fact on the page.
FinCriticalED (HuggingFace TheFinAI/FinCriticalED, Apache-2.0) is a set of real degraded SEC EDGAR financial pages with expert-annotated fact tags: numbers, dates, monetary units, reporting entities, financial concepts. We rendered each page, ran it through paddleocr via our parse router, and checked each gold fact for presence anywhere in the parsed output.
Canonicalization is applied identically to gold and OCR text, tolerant on formatting and strict on meaning: thousands separators and currency symbols are dropped (so 1,200,000 vs 1200000 never counts as an error), but decimal points, minus signs, and % are kept — a shifted decimal or a dropped sign is a critical error and must never be normalized away. Matching is token-boundary substring, so 3.5 never matches inside 13.5.
Result: 7.7% of 491 gold facts appear nowhere in the OCR output, concentrated in plain numbers (8.3%) and reporting entities (8.3%) — the two kinds a financial-document pipeline can least afford to lose.
The obvious counter-move is a confidence threshold: route low-confidence output to human review. We calibrated exactly that — each gold fact paired with the OCR confidence of its containing block (or, when no block contains it, the block where its surrounding context landed, as a proxy for local OCR quality):
| confidence bin | fact survival | n |
|---|---|---|
| [0.95, 0.99) | 0.912 | 80 |
| [0.99, 1.0) | 0.925 | 411 |
Every fact's paired confidence lands in the top two bins, and survival is nearly identical in both. The reliability curve is flat: the minimum confidence achieving ≥95% fact survival does not exist on this data. You cannot buy omission-safety with a confidence floor at any price.
The reason is structural, and it generalizes beyond this dataset: an omitted fact produces no low-confidence block — it produces no block at all. Confidence is a property of emitted text. Omission is the absence of emitted text. A threshold on the former is, by construction, blind to the latter. (Confidence still carries some signal for other failure modes: on scanned pages, blocks overlapping occluded regions like signatures and stamps average 0.893 vs 0.963 elsewhere — real, but far too weak to route on at block granularity.)
If confidence can't see omission, what can? Something that compares what is visibly on the page against what OCR produced. We built the cheapest version that could work — geometric ink coverage, no new model dependency: render the page, Otsu-threshold it to an "ink" mask (dark pixels = visible content), and measure the fraction of ink that falls inside no OCR block's bounding box. That uncovered_ink_ratio sees exactly what confidence cannot: visible content that produced no block.
Validated against both datasets with omission/occlusion ground truth, it splits cleanly into a win and a limitation:
That is an honest partial answer. Region-scale omission — occluded signatures, stamps, logos — is detectable today with geometry alone. Individual-fact omission needs a finer instrument: layout-model coverage, scoring the regions a layout detector finds against the OCR blocks that fill them. We publish the gap rather than papering over it, because the gap is the actual state of the art.
(5.2) vs -5.2) count as different, and a value OCR-split mid-word across two lines counts as lost.git clone <repo> && cd contract-rag && uv sync --extra dev
# One-time: accept the dataset terms on the HuggingFace page (gated auto-approval), hf auth login
uv run python -m contract_rag.eval.fincritical # omission rate + confidence reliability table
uv run python -m contract_rag.eval.coverage # ink-coverage validation on both ground-truth sets
The OCR parses are cached, so re-runs are fast. If your numbers differ materially, open an issue — that is the point of measuring in public.