OpenAI OCR on Base64 Data Report

openai-report-base64.md

OpenAI OCR on Base64 Data Report

URL: https://gist.github.com/monperrus/6cc2679c5f01fc3d95621e0bb431a91a

Tool under test: ~/bin/ocr-openai.py — OpenAI vision model through the Codex ChatGPT login (default model gpt-5.4, --detail original).

Goal

Determine the maximum number of base64 characters that can be put on an A4 page and OCR'd with ocr-openai.py, and compare with the results in https://www.monperrus.net/martin/perfect-ocr-digital-data.

Method

1. A4 pages (210 x 297 mm) generated with ReportLab, DejaVu Sans Mono, 9 mm margins, lines filled edge-to-edge with characters drawn uniformly from the base64 alphabet A-Za-z0-9+/=.
2. PDF rasterized at 400 DPI grayscale with pdftoppm (3308 x 4678 px) — equivalent to a good flatbed scan of a laser-printed page.
3. OCR with ocr-openai.py --detail original.
4. Output compared character-by-character against ground truth (cmp.py): substitutions only; line and column alignment were checked first.

Measured accuracy (gpt-5.4, 400 DPI)

size	chars	wrong chars	CER
5 pt	23,580	8,357	35.4%
6 pt	16,350	1,343	8.2%
7 pt	12,126	596	4.9%
8 pt	9,184	545	5.9%
9 pt	7,300	308	4.2%
10 pt	5,850	209	3.6%
11 pt	4,838	364	7.5% (one-off outlier run)
12 pt	4,050	125	3.1%

One 7 pt page was also scanned at 300 DPI: CER 6.6% vs 4.9% at 400 DPI, so 400 DPI is the better operating point here too.

Font comparison (gpt-5.4, 400 DPI)

Same generator, same seed, fonts substituted (OCR-A from /usr/share/fonts/truetype/ocr-a/OCRA.ttf, FreeMono from freefont, Inconsolata converted OTF->TTF for ReportLab). Note OCR-A and Inconsolata are narrower than DejaVu (500-715/1000 em advance vs 602), so they fit more characters per line at the same point size.

font	size	chars/line x rows	chars	wrong chars	CER
OCR-A	12 pt	63 x 54	3,402	471	13.8%
OCR-A	9 pt	84 x 73	6,132	1,012	16.5%
Inconsolata	12 pt	90 x 54	4,860	262	5.4%
Inconsolata	9 pt	120 x 73	8,760	436	5.0%
FreeMono	12 pt	75 x 54	4,050	100	2.5%
FreeMono	9 pt	100 x 73	7,300	262	3.6%
DejaVu Sans Mono (ref.)	12 pt	75 x 54	4,050	125	3.1%
DejaVu Sans Mono (ref.)	9 pt	100 x 73	7,300	308	4.2%

Ranking by CER: FreeMono < DejaVu Sans Mono < Inconsolata << OCR-A.

Confusion profiles:

font	dominant confusions
OCR-A (9,533 chars, 1,483 errors)	O->0 (125), 6->b (115), l->1 (96), 1->l (53), 0->O (30), Q->@ (27), W->w (24)
Inconsolata (13,619 chars, 698 errors)	l->1 (149), O->0 (66), 0->O (27), 1->l (13), K->k (7), l->I (6)
FreeMono (11,350 chars, 362 errors)	l->1 (87), 0->O (34), O->0 (21), 1->l (17), Z->z (10), g->q (6)

Inconsolata and FreeMono keep the error set within the classic O/0, l/1, case-confusion classes the blog describes.

OCR-A is the only font producing large numbers of non-canonical confusions (6->b, Q->@): designed for magnetic-ink banking reads, it renders ambiguous glyphs for the base64 alphabet — visibly wider letterforms like Q/@ and 6/b overlap.

Structural accuracy was perfect at every size: correct line count, correct characters-per-line, no insertions or deletions anywhere. All errors are single-character substitutions.

Error taxonomy (aggregate over the 9/6/12 pt pages, 27,692 chars)

confusion	count	share
O -> 0	237	13.3%
0 -> O	79	4.4%
l -> 1	65	3.7%
l -> L	46	2.6%
W -> w	35	2.0%
g -> q	21	1.2%
1 -> l	20	1.1%
o -> 0 / o -> O / C -> c / f -> F / k -> K / x -> X / c -> C / j -> i	~70	~4%

Aggregate CER 6.4%; every entry is either a digit/letter confusion (O/0, l/1, o/0) or a case confusion (W/w, C/c, F/f, K/k, X/x) — precisely the two failure classes the blog predicts for base64.

base58 study (gpt-5.4, 400 DPI, DejaVu Sans Mono)

gen.py --alphabet base58 draws from the Bitcoin alphabet 123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz, which deletes the four visually ambiguous symbols 0, O, I, l (and the base64-only +, /, =). This is the blog's "synthesized alphabet" idea, applied to base64's exact failure set.

size	chars	wrong chars	CER (base58)	CER (base64, same page)
12 pt	4,050	10	0.25%	3.1%
9 pt	7,300	210	2.9%	4.2%
6 pt	16,350	2,290	14.0%*	8.2%

*The 6 pt base58 run required a downscaled JPEG upload (upload of the full 4.2 MB PNG hit repeated connection resets), which itself degraded accuracy; the 6 pt numbers are not directly comparable between alphabets.

Confusion profile collapses to case confusion almost exclusively:

• 12 pt (10 errors): W->w (5), K->k (2), V->v (1), g->q (1), m->n (1) — zero digit/letter confusions.
• 9 pt (210 errors): W->w, g->q/q->g, K->k/k->K, Z->z, V->v, J->j — case confusions plus 1->l (3, i.e. the model hallucinating the excluded l) and o->0 (2, hallucinating the excluded 0).
• 6 pt (2,290 errors): same case set (W->w, g->q, U->u, k->K, V->v, Z->z, J->j).

Findings.

1. base58 cuts CER by ~12x at 12 pt (3.1% -> 0.25%) and ~1.5x at 9 pt, and eliminates the entire O/0 digit/letter class that dominates base64 errors. This confirms the blog's prediction that removing confusable symbols from the alphabet is the highest-leverage fix — more leverage than any font or DPI tuning measured here.
2. Case confusion is the new floor. With 0/O/I/l gone, every remaining error is upper/lower case (W/w, K/k, V/v, g/q, Z/z, J/j) or a hallucination of an excluded glyph (1->l, o->0). base58 cannot reach 0% CER as-is; a case-folding variant (e.g. z-base-32 style, or base58 forced to one case at 50% information loss) would be required, which is exactly why the blog landed on lowercase-only base16 and BIP39 for 100% accuracy.
3. Payload trade-off: base58 keeps 58/64 = 90.6% of base64's information density, so the 9 pt page holds 6,615 decoded bytes vs 5,475 — base58 is strictly better per page here: fewer errors and more payload.
4. Repairability: with the error set reduced to ~12 ordered case pairs, the blog's per-line CRC32+Damm repair converges far faster — each line has ~3 suspect characters with ~2 candidate fixes each, well within brute-force per-line search.

Blog reference numbers (A4, GOCR/Tesseract era):

encoding	capacity	accuracy
base16 hex, GOCR, Inconsolata 12 pt, 400 DPI	~2.5 kB	100%
BIP39, Tesseract, Inconsolata 12 pt, 300 DPI	~1 kB	100%
base64 + per-line CRC32/Damm repair, Inconsolata 11 pt	~4.2 kB	100% after repair
base64 on microfilm, 24 pt (Schilke & Rauber)	1.8 kB	—

1. Density is no longer the binding constraint. The blog's setups top out at 4.2 kB/A4. gpt-5.4 keeps perfect line and column alignment on pages 3x denser (16,350 chars) — the page layout is never misread, which GOCR and Tesseract could not do at these sizes.
2. The alphabet still is. Raw CER never reaches zero; errors are exactly the O/0, l/1, l/L, W/w, c/C set the blog calls out. Base64 remains "a nightmare" for LLM vision as well.
3. Convergent payload. 9-12 pt OpenAI OCR gives 4,050-7,300 chars (3.0-5.5 kB) at 3-4% CER — the same order as the blog's 4.2 kB repairable base64, but obtained with an off-the-shelf cloud model instead of a tuned engine/font/DPI/font-size pipeline.
4. Repairability extends further. Because alignment is preserved even at 6 pt (8.2% CER, all substitutions), the blog's per-line CRC32 + Damm checksum repair architecture becomes applicable to pages up to ~16k characters — pages that were unrecoverable with 2010s OCR. With ~90 chars per line and ~3-8 substitutions per line, brute-force confusion-pair repair per line is tractable.
5. Unchanged conclusion: for guaranteed 0% CER without any repair layer, encodings without confusable symbols (base16 lowercase, BIP39) remain the right answer; the confusability of base64 is not fixed by better OCR models. base58 (see above) is the measured intermediate point: 12x fewer errors than base64 at 12 pt, 90% of its density, but case confusion still blocks 100% raw accuracy.

Reproduce

python3 gen.py --size 6 --cols 0 --pdf p6.pdf --txt p6.txt
pdftoppm -r 400 -gray -png p6.pdf s6
~/bin/ocr-openai.py s6-1.png > out.txt
python3 cmp.py p6.txt out.txt

# base58 variant
python3 gen.py --alphabet base58 --size 9 --cols 0 --pdf b9.pdf --txt b9.txt