Structure

Structuring an Encyclopaedia: Entries, Senses and Cross-References

The defining mistake in reference digitisation is treating a dictionary like a book. A book is a sequence of pages. A dictionary is a structured dataset that was printed as a sequence of pages because print was the only delivery mechanism available. Recover the structure and the collection becomes queryable, linkable and repurposable. Recover only the pages and you have a very large PDF.

The entry is the unit, not the page

Start by deciding what your atomic record is, and be strict about it. For a dictionary it is the entry, keyed on the headword. For a biographical encyclopaedia it is the person. For a gazetteer it is the place. Everything else — page images, source citations, illustrations — attaches to that record.

This has an immediate practical consequence: an entry can span a page break, and two entries can share a page. Any model built on page identity will fight this forever. Model entries as first-class records that reference the page images they appear on, and the awkwardness disappears.

Headwords are not unique, and pretending otherwise hurts

Serious reference works are full of homographs. Two entirely unrelated entries can share a headword, and both may be correct. If the headword is your identifier, the second one will overwrite the first, silently, somewhere in the middle of a bulk import.

Give every entry an opaque internal identifier that has no meaning and never changes. The headword becomes an attribute, not a key. This also solves the related problem of headwords that need correcting later — a typo in a headword should not change the identity of the record or break every link pointing at it.

Senses, and knowing when to stop

Within an entry, most reference works subdivide: numbered senses in a dictionary, sections in an encyclopaedic article, sub-entries under a family name. Modelling these explicitly pays off the moment anyone wants to link to one.

There is a limit, though, and it is worth naming. Full lexicographic markup — every etymology component, every grammatical label, every usage note tagged individually — is enormously expensive and is only worth it if somebody will query those fields. For most projects the honest answer is a three-level model: entry, sense, and richly-formatted body text within the sense. Deeper structure can be added later to the parts that turn out to need it; it cannot easily be added to nothing.

Cross-references are the hardest part

Reference works are dense with internal pointers: see, see also, compare, q.v. These are the connective tissue that makes a reference work more than an alphabetical list, and they are where digitisation projects most often fail quietly.

Three problems recur.

• The target is a string, not a link. The printed work says see followed by a headword, and resolving that to a record requires a lookup that may be ambiguous — the homograph problem again.
• The target may not exist. Printed reference works contain dangling references. They pointed at an entry that was cut before publication, or a sibling volume that was never issued. Your data will faithfully reproduce these, and your link checker will report them as your bugs.
• Reciprocity is not automatic. If A points to B, B usually does not point back. Whether you generate reciprocal links is an editorial decision that changes how the work reads, and it should be made deliberately and recorded.

The workable approach is to store cross-references in both forms: the literal text as printed, and a resolved identifier where resolution succeeded. Never discard the literal text. When resolution improves later, you can re-run it; when it fails, you can still display what the book actually said.

Controlled vocabularies save you from yourself

Any field with a small set of legal values — language, subject area, entry type, place type — should draw from an explicit list rather than free text. Free text fields accumulate variants: the same subject spelled three ways, capitalised two ways, abbreviated once. By the time anyone notices, correcting it is a data migration.

Where an established vocabulary exists, borrow it rather than inventing one. The Getty vocabularies cover art, architecture and geography; Library of Congress linked data services cover subjects and names. Borrowed vocabularies also make your data comparable with somebody else's, which is the whole argument for them.

Model for the reader, then for the machine

A last word on priorities. It is easy to design a beautiful schema that no reader benefits from. Before adding a level of structure, ask what a person will be able to do because it exists — jump to a sense, filter by region, follow a reference. If the answer is nothing concrete, it is complexity with no payer.

Structure interacts directly with description; see metadata and catalogues for the layer above. If your material is non-Latin, read character encoding first — sort order and headword identity both depend on it.