Capture

Scanning and OCR: Why Dictionaries Are Harder Than Novels

Optical character recognition is usually described with a single accuracy figure, which is close to meaningless for reference material. The same engine that reads a novel almost perfectly will struggle badly with a gazetteer, and the reason is not the letterforms — it is the layout, the density and the vocabulary.

Layout analysis is the actual hard part

Before recognising a single character, the software must work out the structure of the page: where the text blocks are, what order to read them in, which lines belong together, what is a heading and what is a running header.

On a novel this is trivial — one block, top to bottom. On reference material it is the dominant difficulty. Multiple narrow columns, hanging indents, entries that begin mid-column, tabular matter, marginal notes, footnote apparatus in smaller type, and running heads that look like content. When layout analysis fails, the characters may be read perfectly and assembled into nonsense: two columns interleaved line by line, which is far worse than an honest failure because it is not obviously broken.

Evaluate layout handling before evaluating character accuracy. Run your candidate engines over the ugliest spread in the collection and read the output order. That test discriminates between tools far more sharply than any published accuracy figure.

The vocabulary problem

Modern recognition engines lean on language models — knowing what words are likely improves the reading of ambiguous glyphs enormously. Reference works systematically defeat this. They are full of exactly the material a language model finds improbable: proper nouns, place names, personal names, taxonomic Latin, abbreviations, symbols and numerals.

Worse, the language model does not merely fail to help; it actively harms, by correcting unusual-but-correct strings into common-but-wrong ones. A rare place name becomes a common word. A catalogue number acquires a plausible digit. These errors are invisible to a spell check because the output is a real word.

Where an engine supports a custom dictionary or word list, supplying the collection's own vocabulary — the headword list, the place names, the abbreviations — is the single highest-return intervention available. It converts the language model from a liability into an asset.

Mixed scripts and small type

Reference works mix scripts routinely: an etymology in Greek inside a Latin-script entry, a Cyrillic transliteration, phonetic notation. Engines configured for one language will mangle the others silently. Where a work is systematically bilingual it is usually better to recognise it twice with different configurations and merge, than to find one compromise setting that is mediocre at both.

Small type is the other systematic problem. The abbreviations, cross-reference apparatus and index entries that carry much of a reference work's connective value are set smallest and recognise worst. This is the direct link back to capture: if the smallest type was not captured at sufficient resolution, no engine will rescue it. See digitising reference books for setting resolution from the worst case rather than the average.

Be honest about accuracy

A single accuracy percentage over a whole collection hides everything that matters. Two figures are more useful.

• Character accuracy by page difficulty. Sample the clean pages and the worst pages separately. The gap is usually enormous, and it tells you where correction effort belongs.
• Field-level correctness. For structured material, what proportion of headwords are exactly right? Of coordinates? A collection with 99% character accuracy can still have a materially wrong headword on many entries, because headwords are short and every error lands in one.

Publish the figures. A collection that states its recognition quality is far more useful than one implying perfection, because readers can calibrate. It also makes correction contributions possible.

Correction strategy

Full manual correction of a large reference work is rarely affordable, so the question is where to spend.

Correct the structural fields first — headwords, identifiers, cross-reference targets, coordinates. These are short, high-value, and errors in them break navigation and identity rather than merely reading. Body text can tolerate residual noise; a wrong headword makes an entry unfindable.

Keep the recognised text alongside the page image rather than replacing it, so a reader who hits an obvious error can check the original. This also means imperfect text can be published immediately instead of waiting for a correction pass that may never be funded.

Tools

Tesseract remains the standard open engine, supports custom word lists and many scripts, and is worth benchmarking against any commercial option on your own worst pages rather than on anyone's sample set. Whatever you choose, keep the page images — engines improve, and a collection that retained its masters can simply re-run recognition years later and get a better result for nothing.