I have a need to parse GEDCOM files to extract:
For all Individuals: their (primary) surname, given names, plus a unique ID, and life dates (birth/baptism/christening - death/burial)
For all mentioned Places: as many comma-delimited fields as required identifying the place
For all Repositories: Name, Address and Website URL
For all Sources: Title, Type, and Actual Text
What resources exist to do this parsing? Is there an algorithm I can code for each element?
I'm not aware of a language-independent documented algorithm for extracting data from a GEDCOM.
There are GEDCOM parsers for most programming languages, of varying vintage.
They mostly don't follow any documented process to extract data (other than what's in their source code!), they don't have any common output format, and they each have their own errors and omissions.
The inconsistencies are not (entirely) the fault of each parser, there are ambiguities in the specification(s), and many vendor-specific extensions.
Few parsers have much in the way of reporting (such as "list all places"), so to use them you need to understand GEDCOM at quite a low level.
For example, there is an online parser using the Java parser by Dallan Quass.
This turns GEDCOM into JSON, but doesn't add much intelligence to the output. For example a placename in the GEDCOM like:
1 PLAC
2 FORM City, County, State, Country
is parsed as:
{
"tag": "PLAC",
"children": [
{
"tag": "FORM",
"value": "City, County, State, Country"
}
]
}
It's debatable whether that's any easier to deal with (but the parser does handle the CONT and CONC lines well). This parser is better documented that most, with a list of what real-world extensions to GEDCOM it uses.
I should also mention Tom Wetmore's Lifelines here. This venerable program from the early days of computerised genealogy appears to be capable of being scripted to extract information from GEDCOMs (possibly not the more recent ones) but has a steep learning curve (too steep for me!), basically a new language to learn.
Taking your request 2: "extract places":
This initially looks trivial - grep ' PLAC ' my.ged would give you all PLAC lines. It would work for a large number of the GEDCOMs out there.
But as Louis Kessler points out, it's possible for the place to be described in a few other ways as well, by different software. Maybe there is a parser out there than can cope with all those options, but a comprehensive test suite would be the only way to find out, given the lack of documentation for most.
Sources are equally a mess, with a wide variety of alternative implementations that have to be considered in any parser. And most don't.
Understanding the varied GEDCOM formats is a hard problem that most of the simple parsers are not very good at.
The genealogy programs have more users so more reason to get it right. At present I would say that importing the GEDCOM into a program such as Gramps and using the reporting features there would be the simplest way to work around the problem. It won't produce perfect results all the time, but it's more likely to get it right than tackling the problem at a lower level.
You could take a pre-processor such as the Java parser from Dallan Quass mentioned above, and then with a new program (or ideally a webservice) add intelligence to the output with linkages, validation and reports. But that starts to look very much like yet another genealogy program.
If you're open to using java, gedcom4j is a library that will read GEDCOM 5.5 and 5.5.1 files into objects that you can work with, and can write the objects back out as GEDCOM files as well. It can handle the problem you describe.
There is also Gedxlate available free here. It doesn't do the parsing but does extract to Excel where you could use sorting / pivot tables / data analysis to analyse your data.
Unfortunately it hasn't been updated in a while but it may do what you need, I use it regularly for putting a subset of my data on line in a searchable table.
There is a Python parser for GEDCOM 5.5 format, and a Python module for parsing, analyzing, and manipulating GEDCOM files, available. The latter is based on an earlier version of the former and its use has been blogged about.
About a year ago I contacted the original author and started to use his code but realised that I needed more fluency in Python before I would be able to take full advantage of it. Since then, further work with it has been on the back burner for me, but I do plan to get back to it sometime.
It depends on how you look at it. If your primary goal is extracting the information mentioned in your question, you can read the GEDCOM into a program like Gramps, and create CSV files with the data that you want from that. I mention Gramps because I use and develop that myself, and because it can export most views as CSVs. This means that you can export most of the things you want right from the person, location, and repository views. Getting source texts is more difficult, because sometimes source texts are attached to GEDCOM SOUR objects, and sometimes they're attached to the citation objects instead. These objects appear as different views in Gramps, and both views don't include the texts themselves, which Gramps sees as notes, that have a view of their own.
You can also use RootsMagic to read the GEDCOM file, and then extract the information from RootsMagic's database, which is in sqlite format. Its structure is pretty well documentend on an independent wiki, and because it's sqlite, you can easily write your own SQL statements to extract the information that you need. This is assuming that writing SQL is easier than expanding any of the GEDCOM parsers mentioned on Tamura's site.
There is something called "The GEDCOM parser library", described as "a C library that provides an API to applications to parse, process and write arbitrary genealogy files in the standard GEDCOM format".
This reference came from Tim Forsythe's blog on "VGedX: The GEDCOM Validator".
Don't ask me any more about this... I was a mainframe programmer - I can't even spell "C"!
PS - just for clarity, that last bit was British humour! Allegedly.
I recently discovered Familienbande (English Translated Version) while investigating this other thread, which is a FREE German Genealogy program available for Windows, Mac OSX, and Linux. It has localization / Language packs in many languages, including English which make the program usable though the documentation and some dialogs are still in German.
Similar to GEDScape it has some some export functions and can import and export and manipulate GEDCOM files using its template function to basically parse and create custom text, postscript and reports. Including exporting to CSV.
Once exported to CSV I found I was then able to manipulate and use tools in R, Excel, and other tools.
If you save your GEDCOM in XML format, either GEDCOM X or your application's own XML, you can immediately use any of the large amount of XML software to do extractions, lookups, rearrangements, etc. This is IMHO by far the simplest and most reliable way to do the job.
For simple ad-hoc enquiries, the LT-XML2 suite of programs is often enough; or you can write more complex queries in XSLT or XQuery.
gedcomq (https://github.com/elliotchance/gedcom) is a CLI tool and query language for GEDCOM files that can output in a variety of formats.
Disclaimer: I built this.
