Statistical Genealogy, or Knowing what is an extraordinary claim

Question

It is often said in genealogy (or any field of research, really) that “extraordinary claims require extraordinary evidence”. Fair enough. But how does one know what is an extraordinary claim? We can extrapolate from our own experience and society, but our ancestors often lived in very different times and circumstances. What might seem extraordinary or strange to us might have been quite common to them and vice versa.

I'm a moderator on the Mathematica.SE site, so I’m aware of the social network analysis capabilities in Mathematica, even though I’ve never really used them. They, and a recent post on Stephen Wolfram’s blog got me thinking about the genealogical equivalent.

The big sites like Ancestry.com, Familysearch.org and FindMyPast.co.uk have all collected literally millions of trees and built up profiles of tens of millions of individuals and families. It seems obvious that one could build up statistical statements about what was most common historically, and then people could compare their current hypotheses about their own ancestors to these statements to help decide if they are making “extraordinary claims”. I’ll give some examples to show what I mean. Bear in mind that the following are all completely made up, but are the sort of thing I had in mind:

15% of men and 16% of women born in 18th century England married more than once; only 2% of either men or women married more than twice. ( ok so my great^4 grandmother probably didn't marry five times and those are different Alice Smiths... )

Or:

94% of people who were not nobility born in Kent between 1650 and 1800 married someone born within twenty miles of them. Less than 1% married someone born more than 50 miles away. ( ok so I can probably rule out that baptism from Lancashire as being for my great^5 grandfather who married and died in Dorset...)

Or:

Only 10% of men born in the 1700s in England who subsequently married did so for the first time after they turned 30, and if their parents married before the age of 25, only 3% married after the age of 30. ( hmm, so my great^5 uncle is probably that guy, not that guy who married eight years later. So his wife is most likely to be this woman not that one...)

So my question is: has anyone done this, even over a restricted field such as a particular century or country? Surely there is a statistically-minded Mormon over at FamilySearch or wherever who has thought of doing this? While the exceptional cases clearly can still have happened, it would surely help people formulate their hypotheses if they had an idea how likely or unlikely certain outcomes might have been.

EDIT

Lots of good responses in the answers and comments. I agree that the data are very dirty. But I'm an economist and thus used to dealing with messy, poorly measured data. The question is whether the errors in the data are enough to materially skew the kinds of statistical statements I have in mind. I can imagine that there would be some skew away from correct conclusions that happened to be unusual (the thrice-married woman, the 17th century couple born 100 miles apart etc). But for a lot of things, the errors in people’s trees would cancel each other out. I think this is the difference between a statistical approach and an historical approach.

Answer 1

An example of a study of probable outcomes is: Probert, Rebecca. Marriage Law for Genealogists: the definitive guide.( Kenilworth: Takeaway (Publishing), 2012.). It is based in part on detailed analysis of primary source data about marriage for "entire communities and cohorts" in England, and addresses e.g. :

• How many couples lived together without marrying, based on analysis of illegitimacy rates in the 18th and 19th century (which peaked at 7% in the 1860s).
• what proportion of couples from a single county married in the home parish of one of the couple (41% between 1700 and 1751)

My personal opinion is that the data in trees on FamilySearch, Ancestry etc. is too poor to be a valid basis for statistic analysis.

Answer 2

Without wishing to pre-empt a (more expert) contribution from a statistically-minded Mormon (sic), I suggest that your idea (while superficially very attractive) demands that four things come together to make it work.

1. Analytical tools capable of crunching the relevant free-form data
2. Sufficiently extensive big data stores "open" to exploitation
3. Genealogical knowledge of the types of hypothesis that need a reality check
4. Confidence in the "cleanliness" of the data.

As you suggest, Wolfram has shown us that 1 can be done. All the advertisements claim that 2 is a reality. Forums such as SE could provide 3.

But, IMHO, we stumble at 4. How can I have any confidence in your claim that great grandma did not marry 7 times when your "evidence" is based on a dataset in which grandma is duplicated at least a dozen times. We know from the opening up of FS Family Tree that this situation is the norm rather than an exception. Lest anyone claim that Ancestry does better, I have one word for you - Mocavo.

Last century, big-iron number crunchers encapsulated the problem in the acronym GIGO. It will be difficult to get anyone to put in the effort to carry out the analysis or to take the results seriously when your raw material (taken en masse) is so clearly garbage.

Answer 3

The kind of generalizations you've used as examples have been formulated (although usually without statistical rigor) by many researchers working with original sources instead of conclusion trees. Naming, occupation and residence patterns and surname distribution, for example, may be readily determined if the source includes enough detail. One-place studies are invaluble for looking at this kind of data across multiple source types. There have been articles published in sociological journals that do include the math, but the topics tend to be less helpful to the average genealogist.

Answer 4

I googled "statistical genealogy" - in the expectation of no hits and found this question.

LitvakSIG, the Lithuanian-Jewish research group (http://www.litvaksig.org), publishes data to paid-up contributors in the form of excel files: and this makes this sort of study straightforward for a particular locality. One can even do some sorts of longitudinal time studies for those locales with enough data.

So, in an amateur way, I have been posting over the last 6 years about Jewish marriages, deaths and population growth on one district in north east Lithuania. Do please drop by at http://zarasai.blogspot.co.uk/search/label/Statistical%20genealogy.

The posts include:

• a quick look at population growth in one shtetl over the period 1784 to 1887 which shows a nice log curve with population doubling about every 16 years.

• age at death statistics for the period 1922 to 1939 for two districts of interwar Lithuania - comparison to Swiss mortality for 1928 with good eye-ball correlation - odd dip in mortality at ages 71 to 75 - possible use of Swiss data for earlier period to gain possible insights into Litvak deaths at that time.

• analysis of marriage data for 1877 to 1915 which shows a steady increase in age at marriage of about 1 year every decade to about 1900 and then a sudden drop - about when mass emigration accelerated. Some analysis of the geographical dispersion of spouses with about 70% from nearby locations within 50 km, but some from Georgia and Dagestan - about 2,800 km away by road.

20/21 December 2014

Answer 5

What a fascinating question and discussion. I don't believe that dirty data is an issue here. Any applied research has to deal with effects of data quality and genealogical data doesn't appear to be obviously special in its characteristics.

One of my frustrations with the current genealogical tool sets is that they don't make it especially easy to, for example "chart a time series of %illegitimate births by time hycketed every ten years. When I look at other people's ancestry public trees it appears that there are a good many trees of size 10,000 but fat far fewer trees that are 40k in size

Have you looked at the British names and world surname GIS projects? These are a useful resource that offer statistical insight into specific surnames. When I look at RootsMagic or Mac Family Tree I can produce reports that list improbable facts. It would be great to see these listed in probability order.

I'm wondering what the smallest, lowest scope example of the functionality you describe would be?