1

So I'm having some difficulty using ancestry's search engine to locate someone in a Massachusetts Index of Marriages. Rather than trying to problem solve that directly, it occurred to me that having some idea of the process used by services like Ancestry, Family Search, NEHGS etc to populate their databases would be useful to troubleshoot problems in searching those databases.

(I am using the Massachusetts Index only as an example).

Image of Index page from Ancestry.com

Ancestry notes the following relative to this image: Ancestry.com notation of source of the image

At this point, my presumption is that the image shown is (part of a page) of a document originally produced by the State of Massachusetts. Furthermore, that image was either taken by NEHGS or provided to NEHGS by the State.

Now, with the image in hand, the process of getting the information from the image into a computer database (so that the information is now searchable) begins.

At this point I'm assuming that the image is (most commonly) scanned and "read" by a OCR (optical character reader) thereby making it possible to construct or populate a searchable database. (The alternative would be a human reads the image and keyboards in all the information - reasonable for script, but probably considered unnecessary for typeset).

Assuming that my presumptions are correct above, my question at this point is: do each of the organizations (eg. Ancestry, Family Search, NEGHS etc) independently scan and OCR the original image or do they share the OCR files with each other.

My reason for asking this is that I've noticed that (a) OCR reading is not always accurate and (b) database searches (of the same original documents) are sometimes different between the organizations.

(I have chosen to begin my understanding of the "indexing" process with printed (typeset) original information presented in column format. Script information is a whole 'nother thing)

  • 2
    The search algorithms between the various sites/companies are also different, which would help account for differences in results from searching the same records at various sites. – Leah Worster Apr 17 '19 at 17:28
  • 1
    The answer to "Do they ... or do they ...?" is probably "Yes". Unless the companies make a public statement or brand their stuff with the originator, it's impossible to know. Commercially confidential, they'll say. My guess is that every variation that you can think of, will be found somewhere and while some companies may have favourite methods, they probably won't tell. – AdrianB38 Apr 17 '19 at 18:09
  • 1
    I wouldn't assume OCR is standard for reading typescript either. Those ditto marks and omitted surnames (because they match the line above) are not trivial to deal with. If the layout is consistent through the pages, it might be feasible to OCR it. I only know that Ancestry's OCR of City Directories can be appalling because it doesn't recognize where the line starts. FindMyPast used OCR on its electoral registers but because of the many layouts seems to only tell you that the individual words in a name are present somewhere on the page, not always together. It's not easy. – AdrianB38 Apr 17 '19 at 18:18
  • 1
    One last thing - indexes might be shared and start out the same but if each company accepts corrections, they will drift apart. – AdrianB38 Apr 17 '19 at 18:21
  • 1
    As you can see, FS indexing is human volunteers looking at (digital) images and filling out fields. The old method was two indexers and an arbitrator (if the two indexers filled out a field differently); the new method is a single indexer and a reviewer. Stuff gets (badly) mis-indexed either way. – JPmiaou Apr 18 '19 at 14:27
3

Let's start with the original source material which you could use if you were at the Massachusetts Archives.

The following quotes are excerpted from the article Massachusetts Vital Records in the RootsWeb Wiki (formerly the Ancestry Wiki) which is a wiki adaptation of Ancestry's Red Book. The article intro says:

This entry was originally written by Alice Eichholz, Ph.D., CG, for Red Book: American State, County, and Town Sources.

Beginning in 1841 the state mandated that a copy of each event recorded in a town or city be sent to the Secretary of the Commonwealth, which means that two sources exist for each event after that date: the town (or city) and the state. However, some towns were not in compliance until the late 1840s. At present, the indexes for 1841 to 1910 are in bound, ledger-style books, arranged in five-year periods, except for the first, which covers 1841 through 1850. The records are available at Massachusetts Archives. (emphasis mine)

As of 1 January 1896, the Massachusetts Registry of Vital Records and Statistics, 150 Mt. Vernon St., Dorchester, MA 02125-3105 is the repository for copies of town or city recorded vital records. The facility is open to the public, but marriage and birth records may be restricted. As with the bound ledger-style volumes for the 1841 to 1910 period, indexes continue in five-year periods, separated into births, marriages, and deaths. Records and indexes are transferred to the Massachusetts Archives every five years. (emphasis mine)

The 1841 to 1910 vital records ledger books and indexes for the state are open to the public and available on microfilm and in book form at the New England Historic Genealogical Society and through the Family History Library (FHL).

Ancestry's About the Database for Massachusetts, Marriage Index, 1901-1955 and 1966-1970 cites:

Original data: Department of Public Health, Registry of Vital Records and Statistics. Massachusetts Vital Records Index to Marriages [1916–1970]. Volumes 76–166, 192– 207. Facsimile edition. Boston, MA: New England Historic Genealogical Society, Boston, Massachusetts.

ABEBooks' website defines facsimile edition as follows: "A facsimile edition is when a publisher recreates, with perfect detail, a particular publication." The intent is to produce a volume which allows the reader to have as much as possible the experience of reading the original work. Some of the techniques that have been used to make facsimile editions are listed in Wikipedia's article Facsimile; most recently they are likely to have been made with some kind of photographic technique.

So far the process of creation looks like this:

  1. Towns submit copies of their records to the state.
  2. Some state agency indexes the records. We saw from the examples in the question Understanding how to use Massachusetts Marriage Index that there is a stamped number on the page which is not original to the page number on the copy register held by the state.
  3. The state produces the five-year index volumes.
  4. NEHGS produces a fascimilie edition, intended to look as much like the original index volumes as possible, most likely by some kind of photography or scanning of the state's volumes.
  5. Someone, probably Ancestry, takes photos or scans of the bound books and produces its computer index. Ancestry's About the Database section on the database search page says (emphasis mine) "The index for this database was generated using text recognition software." In other words, the underlying text that we see when we are in the viewer and open up Ancestry's index was gathered up by a computer using what we call OCR (optical character recognition), and not entered by a human.

Sometimes Findmypast, FamilySearch, Ancestry, etc. will say whether or not they are sharing a common index, usually with a statement saying 'this index was provided by' followed by the name of the entity who made it. But in the majority of cases, they don't say. Search engines differ from site to site, so I always search what appears to the 'the same' database at all sites so I can compare results and look for hits that show up at one site but not another.

Kenneth Marks' The Ancestor Hunt, a resource for finding online newspapers and more, has several articles on dealing with OCR problems, since OCR is also used for historical newspaper pages. His article 8 Ways to Overcome OCR Errors when Searching Newspapers is a good place to start, and has links to other articles such as the Project Gutenberg Scanning FAQ. One of his quick reference guides, The Best Way to Find Hidden Newspaper Articles, has tips about which letter pairs are often confused in OCRed text.

| improve this answer | |
  • Can you expand on point #5? Possibly where you found this discussion "About the Database". Frankly, I am being hampered by my previous (years ago) experience about how to populate databases particularly when the original or image of the original) is in neat columnar format. I agree that the "search engines" will differ, but the database being searched should remain consistent (except for corrections). The question then becomes: Does each provider (Ancestry etc) scan/OCR the original image to create their own unique database ? – BobE Apr 17 '19 at 22:03
  • just so I'm clear when I say, their own unique database, I'm referring to the data base associated with just the original document, in our example case which just happens to be and index. – BobE Apr 17 '19 at 22:07
  • @BobE to find "About the Datbase" for an Ancestry database, go to the search page / browse page for the individual database and scroll down the page. You can reach that page for any database from Ancestry's Card Catalog. search.ancestry.com/search/… – Jan Murphy Apr 17 '19 at 23:16
  • @BobE In this case "the original document" just happens to be a bound book, not a document. – Jan Murphy Apr 17 '19 at 23:28
  • 1
    Hi Jan, found your "about this database" reference. I'm thinking that the folks at Ancestry are using the word "index" incorrectly. It's a technicality, but text recognition (OCR) does not index a database. Text Recognition can be used to create the database, indexing is how the data is subsequently organized to provide efficient searches. That is to say, a programmer or data specialist sets up indexing, the OCR provides the "things" to be indexed. – BobE Apr 18 '19 at 2:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.