Understanding the indexing process used to produce records searches?

Question

So I'm having some difficulty using ancestry's search engine to locate someone in a Massachusetts Index of Marriages. Rather than trying to problem solve that directly, it occurred to me that having some idea of the process used by services like Ancestry, Family Search, NEHGS etc to populate their databases would be useful to troubleshoot problems in searching those databases.

(I am using the Massachusetts Index only as an example).

Ancestry notes the following relative to this image:

At this point, my presumption is that the image shown is (part of a page) of a document originally produced by the State of Massachusetts. Furthermore, that image was either taken by NEHGS or provided to NEHGS by the State.

Now, with the image in hand, the process of getting the information from the image into a computer database (so that the information is now searchable) begins.

At this point I'm assuming that the image is (most commonly) scanned and "read" by a OCR (optical character reader) thereby making it possible to construct or populate a searchable database. (The alternative would be a human reads the image and keyboards in all the information - reasonable for script, but probably considered unnecessary for typeset).

Assuming that my presumptions are correct above, my question at this point is: do each of the organizations (eg. Ancestry, Family Search, NEGHS etc) independently scan and OCR the original image or do they share the OCR files with each other.

My reason for asking this is that I've noticed that (a) OCR reading is not always accurate and (b) database searches (of the same original documents) are sometimes different between the organizations.

(I have chosen to begin my understanding of the "indexing" process with printed (typeset) original information presented in column format. Script information is a whole 'nother thing)

Answer 1

Let's start with the original source material which you could use if you were at the Massachusetts Archives.

The following quotes are excerpted from the article Massachusetts Vital Records in the RootsWeb Wiki (formerly the Ancestry Wiki) which is a wiki adaptation of Ancestry's Red Book. The article intro says:

This entry was originally written by Alice Eichholz, Ph.D., CG, for Red Book: American State, County, and Town Sources.

Beginning in 1841 the state mandated that a copy of each event recorded in a town or city be sent to the Secretary of the Commonwealth, which means that two sources exist for each event after that date: the town (or city) and the state. However, some towns were not in compliance until the late 1840s. At present, the indexes for 1841 to 1910 are in bound, ledger-style books, arranged in five-year periods, except for the first, which covers 1841 through 1850. The records are available at Massachusetts Archives. (emphasis mine)

As of 1 January 1896, the Massachusetts Registry of Vital Records and Statistics, 150 Mt. Vernon St., Dorchester, MA 02125-3105 is the repository for copies of town or city recorded vital records. The facility is open to the public, but marriage and birth records may be restricted. As with the bound ledger-style volumes for the 1841 to 1910 period, indexes continue in five-year periods, separated into births, marriages, and deaths. Records and indexes are transferred to the Massachusetts Archives every five years. (emphasis mine)

The 1841 to 1910 vital records ledger books and indexes for the state are open to the public and available on microfilm and in book form at the New England Historic Genealogical Society and through the Family History Library (FHL).

Ancestry's About the Database for Massachusetts, Marriage Index, 1901-1955 and 1966-1970 cites:

Original data: Department of Public Health, Registry of Vital Records and Statistics. Massachusetts Vital Records Index to Marriages [1916–1970]. Volumes 76–166, 192– 207. Facsimile edition. Boston, MA: New England Historic Genealogical Society, Boston, Massachusetts.

ABEBooks' website defines facsimile edition as follows: "A facsimile edition is when a publisher recreates, with perfect detail, a particular publication." The intent is to produce a volume which allows the reader to have as much as possible the experience of reading the original work. Some of the techniques that have been used to make facsimile editions are listed in Wikipedia's article Facsimile; most recently they are likely to have been made with some kind of photographic technique.

So far the process of creation looks like this:

1. Towns submit copies of their records to the state.
2. Some state agency indexes the records. We saw from the examples in the question Understanding how to use Massachusetts Marriage Index that there is a stamped number on the page which is not original to the page number on the copy register held by the state.
3. The state produces the five-year index volumes.
4. NEHGS produces a fascimilie edition, intended to look as much like the original index volumes as possible, most likely by some kind of photography or scanning of the state's volumes.
5. Someone, probably Ancestry, takes photos or scans of the bound books and produces its computer index. Ancestry's About the Database section on the database search page says (emphasis mine) "The index for this database was generated using text recognition software." In other words, the underlying text that we see when we are in the viewer and open up Ancestry's index was gathered up by a computer using what we call OCR (optical character recognition), and not entered by a human.

Sometimes Findmypast, FamilySearch, Ancestry, etc. will say whether or not they are sharing a common index, usually with a statement saying 'this index was provided by' followed by the name of the entity who made it. But in the majority of cases, they don't say. Search engines differ from site to site, so I always search what appears to the 'the same' database at all sites so I can compare results and look for hits that show up at one site but not another.

Kenneth Marks' The Ancestor Hunt, a resource for finding online newspapers and more, has several articles on dealing with OCR problems, since OCR is also used for historical newspaper pages. His article 8 Ways to Overcome OCR Errors when Searching Newspapers is a good place to start, and has links to other articles such as the Project Gutenberg Scanning FAQ. One of his quick reference guides, The Best Way to Find Hidden Newspaper Articles, has tips about which letter pairs are often confused in OCRed text.