TL;DR: FrankenOCR — a pure-Rust, CPU-only wrapper around a 3B-parameter OCR vision-language model, and the subject of a wave of market hype — went onto our own benchmark harness against PaddleOCR, our incumbent scanned-document engine. Result: extraction parity (field-F1 0.280 vs 0.260 on degraded contracts, n=5 — noise), 25–50× slower (55.9s/page vs 1–2s on the same CPU), and a failure mode we consider worse than classic OCR garble: confident structured hallucination on unreadable pages — which our quality score reads as a perfect 1.000. The adapter stays in the repo as an opt-in experiment; the default engine does not change. Every number is a point-in-time measurement (2026-07-08), committed in content/franken_results.toml, harness included.
FrankenOCR has genuinely attractive properties, and pretending otherwise would be its own kind of dishonesty. It ships as a single 5MB binary with zero Python dependencies (the weights, ~3.9GB, download on first run) — a real operations story for locked-down environments. It emits layout classification labels (header / title / text / page_number / image) that our coverage experiments could use. And the hype cycle around it was loud enough that "we didn't look" would have been negligence.
So we did what we do with every engine: wrote an adapter that converges to the same Document IR every other parser emits — one subprocess call per document, markdown output rebuilt into blocks, opt-in behind a FRANKEN_BIN env var, byte-identical router behavior when unset — and put it on the same harness as the incumbent. Same documents, same extractor, same metrics.
Two arms, run 2026-07-08:
| arm | FrankenOCR | PaddleOCR |
|---|---|---|
| speed (12 real Tobacco800 scans, CPU) | 55.9s/page | 1–2s/page |
| extraction, degrade-light (5 CUAD contracts) | field-F1 0.280 | field-F1 0.260 |
| extraction, degrade-shred (unreadable) | 0.000 | 0.000 |
The extraction difference at n=5 is noise: read it as parity. The speed difference is not noise: 25–50× on the same hardware, which at archive-migration scale is the difference between an overnight job and a quarter.
One number in the raw output looked like a FrankenOCR win and is worth debunking ourselves before anyone quotes it: source-attribution accuracy 0.800 vs 0.400. That gap is an artifact of block granularity, not accuracy — FrankenOCR emits ~2.6 page-sized markdown blocks per document vs PaddleOCR's 68–142 line-level blocks, and citing "somewhere in this page" is simply an easier target than citing the right line. A metric that looks better because the evidence got coarser is not better.
On shred — our deliberately unreadable degradation level, where any engine should fail — PaddleOCR fails honestly: it emits visible garble, and our quality score duly craters to 0.730, flagging the document for review.
FrankenOCR fails differently. It emitted fluent, well-formed, entirely invented structure: empty <table> skeletons, and in one case a 50KB repeating "2. 3. 4. …" list loop that took 963s to decode. Our quality score read that output as 1.000 — perfect.
This is the quality-signal blind spot we measured before, in a sharper form. Quality signals score what the engine emitted: garble triggers them; silence evades them; and fluent hallucination actively defeats them — it manufactures exactly the well-formed evidence the signals reward. For a pipeline whose product is sourced, verifiable facts from legal documents, an engine that invents structure on unreadable input is disqualifying at the default position, whatever its throughput or packaging. Garble gets caught. Hallucination gets trusted.
FrankenOCR stays in the repo as an opt-in experiment — the adapter and benchmark harness are committed, and the layout labels may yet earn their keep in coverage experiments. PaddleOCR stays the default scanned-document engine.
The transferable part is not the verdict but the procedure, which costs about a day once per candidate:
git clone <repo> && cd contract-rag && uv sync --extra dev
# FrankenOCR: 5MB binary, MIT — github.com/Dicklesworthstone/franken_ocr (~3.9GB weights on first run)
FRANKEN_BIN=path/to/focr uv run python scripts/benchmark_franken.py
Needs the Tobacco800 scans and a CUAD golden set (never committed — see the repo README); OCR outputs are IR-cached so re-runs are fast. If a newer FrankenOCR version changes these numbers, run it and tell us — the harness exists so that this argument can be had with data.