TL;DR: Every negative result we've published shares one root cause: PaddleOCR silently omits content, and no downstream signal — not confidence, not geometric ink-coverage, not layout-region coverage — reliably catches it. This time we asked a different question: does a modern vision-language OCR model, which reads a page as a whole instead of stitching detection + recognition, actually fix the omission blind spot? We measured two candidates (dots.ocr and DeepSeek-OCR) against PaddleOCR on the same FinCriticalED fact-level ground truth and the same degrade ladder, plus a brand-new hallucination metric for the failure direction a vision model is more prone to. Result: the vision-OCR model wins on nearly every axis we measured — and still fails the pre-registered adoption bar we fixed before running it. All numbers below are point-in-time measurements (2026-07-13, Lambda A100-40GB, vLLM 0.11.2, ~6.2 GPU-hours), committed in content/vision_ocr_results.toml and reproducible with the commands at the end.
The pattern across every prior article in this series is the same shape: the document-level quality score reads near-perfect (0.998 on FinCriticalED) while 7.7% of expert-labeled facts vanish from the OCR output entirely. OCR confidence can't flag it (an omitted fact produces no block, not a low-confidence one). Geometric ink-coverage and layout-region coverage both catch region-scale occlusion (signatures, stamps) but not a single dropped number. Every fix we've tried treats the symptom — PaddleOCR's detection-then-recognition pipeline stitching regions together and dropping some.
A vision-language OCR model reads the whole page in one pass — layout, recognition, and reading order together — so it has no detection-stitching seam to drop text at. Its failure mode should be the opposite: not silent omission but confident invention. Both directions are measurable against ground truth we already had cached, so the outcome was going to be publishable either way: either the VLM materially fixes the omission blind spot, or it's a fifth well-measured negative result.
Same 85 gold pages, 491 expert facts, same omission scoring as the original fincritical run — now run through both vision-OCR candidates:
| engine | overall omission | number-fact omission |
|---|---|---|
| PaddleOCR (baseline) | 7.7% | 8.3% |
| dots.ocr | 5.9% | 0/72 |
| DeepSeek-OCR | 13.2% | 31.9% |
dots.ocr beats the traditional engine on both columns, and the number-fact result is the headline: it dropped none of the 72 number facts — the ones a contract or financial fact-extraction pipeline depends on most. That strength is specific to numbers, not to digit-bearing facts as a whole: dots.ocr still dropped 8.8% of date/temporal facts, a real residual the number-only result doesn't cover. DeepSeek-OCR moves the wrong way — worse than PaddleOCR overall and dramatically worse on numbers, which we attribute to its optical-compression approach being hostile to exactly the facts that matter, compounded by a hard context-length ceiling that is itself a real serving limitation. And the blind spot that started this whole series is engine-independent: document-level quality reads 0.998–1.0 across all three engines, so the quality score cannot tell you which one is actually dropping facts.
FinCriticalED can only measure omission (a gold fact absent from the output). The degrade ladder — clean digital CUAD pages rendered, degraded, and re-OCR'd — gives ground truth in the opposite direction too, because the original clean text is the reference. We added invented_token_ratio: the fraction of OCR output tokens that don't appear anywhere in the original page text, canonicalized so pure formatting (case, thousands separators, currency symbols) never counts, while a misread digit or sign does.
Field-F1 on the same 5-doc, first-3-page slice used in the earlier degrade run:
| level | PaddleOCR F1 | dots.ocr F1 |
|---|---|---|
| light | 0.26 | 0.32 |
| medium | 0.16 | 0.24 |
dots.ocr wins at both levels. The invented-token metric is where the real reframe happens — it separates confident junk from safe failure: at fax, PaddleOCR invents 0.97 of its output tokens while its own quality score still reads 0.92; dots.ocr invents only 0.107 at the same level. At shred, PaddleOCR invents 0.85, dots.ocr invents 0.0 (it returns essentially nothing rather than confabulating), and DeepSeek-OCR invents 0.70 — heavy hallucination while its own quality score reads a blind 1.0. PaddleOCR, the "traditional" engine, is the worst hallucinator we measured, and the quality formula cannot see any of it — this is the same blind spot as the omission story, just in the other direction.
Caveat on the metric itself: on sparse-cover documents — where even the original clean digital text on those 3 pages is nearly empty — the invented-token ratio is inflated identically for every engine, because a short reference text makes almost any output token count as "not in the reference." The light-level absolutes above sit in a range dominated by this reference-bias effect; the per-doc values on dense-text pages are close to zero. Treat the deltas between engines as meaningful and the absolute numbers as reference-bias-sensitive.
One more operational finding shaped the harness itself: run uncapped, both vision-OCR models repetition-loop on badly degraded pages — generating 100k+ junk tokens and taking 20–30 minutes per page before we added a hard generation cap. A production VLM-OCR deployment needs that cap; it is not optional.
Before any of the numbers above existed, we fixed an adoption bar: the vision model becomes the scanned-route default only if its FinCriticalED omission rate is at or below half of PaddleOCR's — 3.85%. dots.ocr's measured omission rate is 5.9%, which is above that bar. Verdict: missed. PaddleOCR stays the default scanned-route engine; the VLM route stays opt-in. Latency backs this up too — 5–22 s/page (A100) on a GPU versus PaddleOCR's 1–2 s/page (CPU) on CPU is a real cost, not a rounding error, for a bar that wasn't even cleared. But the honest picture is not "don't use it": on field-F1, fail-safe degradation behavior, and near-zero number-fact omission, dots.ocr is the strongest candidate we've measured for an opt-in VLM_ENDPOINT. DeepSeek-OCR is not recommended for this vertical on any axis.
The natural next question, given dots.ocr's number-fact strength: use it as a verifier rather than a replacement. Run PaddleOCR as primary, diff its output against dots.ocr's for digit-bearing "critical tokens" (numbers, amounts, dates, percentages) that are missing from the primary — flag the page for human review when they don't match. We pre-registered a bar for this too, before running it: recall ≥ 50% @ false-alarm ≤ 20%.
Measured offline against the same cached FinCriticalED IRs: overall flag-recall 0.184, false-alarm rate 0.152. Verdict: FAIL on the overall bar (recall ≥ 50% @ false-alarm ≤ 20%) — well short of the recall side of it. The honest reason is a design limit, not a bug: 0/27 entity-name omissions were caught, because the digit-token cross-check can never see a pure-text entity name by construction — entity-name omissions are the single largest category, out of 38 total omissions in the dataset. But on the digit-bearing facts the check exists to protect, it works: it caught 6/6 omitted number facts and 0.875 of all digit-bearing omissions, at that same 0.152 false-alarm rate. The dual-engine crosscheck is a usable digit-fact safety net, not a general omission detector — the same "region-scale vs fact-level, geometric-only signals can't see values" lesson from the coverage work, applied to a second OCR engine instead of a geometric ratio.
DeepseekOCR2ForCausalLM architecture had no vLLM build compatible with the rig's CUDA 12.8 driver, so we substituted DeepSeek-OCR v1. The v1 numbers above are not evidence about OCR-2's specific architecture.dirtify.git clone <repo> && cd contract-rag && uv sync --extra dev
# Build the OCR caches first (gated/external datasets — see the OCR-omission article):
uv run python -m contract_rag.eval.fincritical
uv run python -m contract_rag.eval.degrade
# Vision-OCR measurement needs a served VLM endpoint (see scripts/measure_vision_ocr.py's RUNBOOK):
uv run python scripts/measure_vision_ocr.py --model dots
uv run python scripts/measure_vision_ocr.py --model dsocr
# Dual-engine crosscheck is fully offline once both IR caches exist:
uv run python -m contract_rag.eval.crosscheck
If your numbers differ materially, open an issue — negative results only stay useful while they stay true.