Structure

Character Encoding: How Non-Latin Archives Destroy Themselves

An English-language archive can survive decades of sloppy encoding practice because the characters it uses happen to be identical in almost every encoding ever devised. A Cyrillic, Greek, Hebrew or Central European archive has no such protection. It is the same content, the same effort and the same care — and it is structurally far more likely to become unreadable.

Why the risk is asymmetric

Nearly every encoding in common use agrees on the first 128 characters, inherited from ASCII. Unadorned English text is therefore accidentally portable: mislabel it, transcode it wrongly, run it through a careless tool, and it usually emerges intact.

Everything above that range is where encodings disagree, and that is precisely where every non-Latin script lives. The same byte means one letter in one codepage and a different one in another. There is no redundancy in the format to detect the error and no way to recover the original from the bytes alone once a lossy step has occurred.

The practical result is that a great many older regional archives — exactly the kind of subject libraries that are most worth preserving and least likely to have institutional support — are sitting in legacy single-byte codepages, sometimes correctly labelled, often not.

The legacy codepages you will meet

For Cyrillic specifically, several mutually incompatible encodings were in wide use simultaneously, which is worse than one bad standard. Windows-1251 became dominant on the desktop; KOI8-R was long established on Unix systems and in email; ISO-8859-5 was the nominal international standard and the least used of the three; and DOS-era codepage 866 persists in older archives and filenames. A file in any one of them, opened as any other, produces plausible-looking but wrong Cyrillic rather than an obvious error.

That last detail matters enormously. The failure is not loud. Text does not become blank or throw an exception — it becomes different letters. Automated validation passes. Only a reader of the language notices, and by then the mis-transcoded copy may be the only one.

Mojibake, and reading it as a diagnostic

The garbled text usually called mojibake is not random, and its pattern identifies the fault. Cyrillic read as Western European produces a characteristic run of accented Latin capitals; the same text double-encoded through UTF-8 produces a longer, distinctive sequence with recurring leading characters. Recognising the shape tells you which pair of encodings collided, and single-step errors are usually reversible.

Double-encoding — text converted to UTF-8 twice — is the most common serious case and, importantly, is often still reversible, because the transformation is deterministic. What is not reversible is a conversion that passed through a target encoding lacking the characters, replacing them with question marks or a substitution character. Those bytes are gone.

The migration, in order

The safe sequence has four steps and skipping any of them causes trouble.

1. Determine the true encoding of each file. Do not trust a declaration in the markup — the declared encoding and the actual bytes disagree distressingly often. Detect from the bytes, then verify by having someone who reads the language look at a decoded sample.
2. Keep the originals. Copy, never convert in place. If a conversion turns out to be wrong, the untouched originals are the whole recovery plan.
3. Convert to UTF-8 and record what you did. Note the source encoding per file. That note is the difference between a fixable mistake and a mystery.
4. Fix the declarations. A correctly encoded file served with a wrong charset header is still broken in a browser. The HTTP header, the meta declaration and the actual bytes must all agree, and the header wins where they conflict.

The parts everyone forgets

• Filenames. They carry encoding too, and a filesystem copy across platforms can mangle non-Latin names while leaving file contents perfect.
• The database. Connection charset, column charset and stored bytes are three separate settings that can each be wrong independently, producing text that looks right in one client and garbled in another.
• Sort order. Correct alphabetisation is a collation question, not an encoding one. UTF-8 fixes storage; it does not make your headwords sort correctly. Choose a locale-aware collation deliberately.
• Normalisation. The same visible character can have more than one valid byte sequence. Pick a normalisation form and apply it consistently, or identical-looking headwords will fail to match.

Do it before anything else

Encoding sits underneath every other decision. Search, sorting, deduplication and identity all depend on the bytes being right. Migrating encoding after building indexes and identifiers means rebuilding both. The Unicode Consortium publishes the standard and its collation and normalisation annexes, and the W3C internationalisation activity has the clearest practical guidance on declaring encodings correctly on the web.

If you are working through a collection systematically, this belongs before structuring and before recognition — both produce text, and both need somewhere correct to put it.