I somehow assume that there are more information publicly available on the internet about ancestors than google can/does show us. I see this sometimes when new relevant search results come up after I searched again for a known ancestors typing his/hers full name and a location, which was very prominent during his/hers life.

I have heard about the fact that most parts of the internet are not searchable (~90 %), see image below.

  1. Is the deep web relevant to us in terms of genealogical research?

  2. Are there any (open source/commandline) tools, which can access the deep web or did I completely misunderstood the principle of the deep web?

enter image description here


1 Answer 1


Generally, the iceberg analogy, as applied to genealogy, refers to the fact that most records are not digitized and not online. However the "deep web" (as defined on the graphic in the question: information not accessible by surface web crawlers) for genealogy, includes

  • records behind paywalls or requiring registration even to view
  • dynamic webpages, created on the fly, from databases not directly accessible
  • webpages not linked to anything else
  • obscure sites (low ranking)

That means that, although Google may find the homepage of the site, it can't bring up many actual records.

I've noticed, however, that some sites (i.e. Findagrave.com) are becoming more directly accessible, possibly because of better linking of pages within the site, blog postings referencing pages within the sites (giving the searchbots a foothold within the site), or changes in the search engine algorithms.

In the meantime, the answer is to find the sites (by whatever means) and then, use the site's own search process, which is more versatile and tailored to the site, to access the actual information and records.

There are "helper" sites or applications, that can act as intermediaries:

To directly access deeper into genealogy websites:

  • hints and search from within your genealogy software
  • One-Step Webpages by Stephen P. Morse (uses usernames and passwords to the actual record providers to get access)

To supplement search engine indexing:

Your mileage may vary:

  • genealogy toolbars (too limited, in my opinion), example RelativelyCurious Toolbar

  • Mocavo - once free and focused on the world-wide web, now re-directs to FindMyPast and is focused on FindMyPast's collections, which are not free.

  • 1
    This site has lots of relevant genealogy books indexed, and is a good example of a site holding a lot of info that wouldn't come up in a Google search: gengophers.com
    – Jonny Perl
    Jun 9, 2017 at 18:24
  • @JonnyPerl I can't get any results from gengophers - probably my security settings are blocking something critical, but I'm not willing to experiment.
    – bgwiehle
    Jun 9, 2017 at 22:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.