Using git for genealogy collaboration

Question

Are there any existing projects or proposed standards for using git (a source version control system) to collaborate on building genealogical source records and/or conclusions?

Git

Git (best known on hosted sites like github and bitbucket, but available as a command-line or graphical application on most platforms) is a way to store (mostly) text and programming source files, so that changes to individual lines within the files can be made by different people in different places, and merged together with a full audit trail of who changed what (called "blame").

It's also possible to split data off into a branch, make proposed changes to it, and then submit it back to those responsible for the main branch as suggested improvements, which can be accepted or rejected, and merged back into the main branch. So it's not a free-for-all like a wiki, there are ways to have control over data yet still accept improvements.

It seems it would make a good fit for genealogy source data, which is often text-based (perhaps with images attached). It would make it easy for transcriptions to be added to and corrected by many people, and more importantly, see who changed it.

It could also be a way to store assertions, and provenance trails - show how a conclusion was reached, in a format that allows corrections and new data.

Existing projects

There are projects around to store gedcom data in git (example: git-ged), which is one way of handling legacy data, but I'm thinking it would be more useful for newer standards that are source-based, rather than just for conclusion reporting formats like Gedcom.

Many of the new standards like Gedcom-X are XML-based (perhaps with an alternate json representation). Neither XML or (to a lesser extent) json map well to a line-based version control system (because it can be hard to retain the structure when a individual lines are changed by different people at the same time). It's also difficult for a human to edit those formats.

Although gedcom is line based it again has a structure that is easy to mess up with multiple editors, and has other limitations (lots of cross-references that are easy to break).

Extending Git

Git (and github) have been extended to display data in different ways, for example to map the data if it is in geojson (a geographic point/line format), or to show tables if it is in csv. So there could be certain requirements (restrictions) of a text-based genealogy format that would make it possible to generate reports from it. It doesn't have to be directly usable in a program, just something that can be edited by multiple people in a fairly easy way.

CSV

For source data (lists of events, simple transcriptions) it's maybe simplest just to stick to csv (comma separated variables) files, that can be easily imported (and exported) by most software.

It would be good to have a csv description language with a genealogy vocabulary that describes the data. The simple data format is one such way to describe csv files. Has anybody been working on marking up genealogy source files with a system like this?

Open Data

Much of this assumes that the source data (events) are in a format which is shareable (and so editable by many). Unfortunately very little data currently has suitable licenses, but that is changing.

Answer 1

You already mentioned git-ged, which is really the only thing at github that comes close to what it seems you want.

Dovy Paukstys added a git entry for SourceTemplates two years ago, but you won't find anything there.

Git is for real geeks and/or programmers, and most genealogists aren't up to snuff to use it.

The tool for most genealogists that is most like what you seem to be asking for (with audit trails, merging, branches, etc.) would be a wiki. They vary in the number of features they have.

WikiTree is the most popular, but there's a whole bunch of others such as these listed at GenSoftReviews.

Some of the web-based programs create full audit trails and allow collaboration such as PhpGedView which is open source, and programs like that might fit many of your requirements.

Answer 2

I realize this is a bit of a ramble, and probably not the answer you are looking for. There is something of real value in the question you are asking, and a solution (by extension) would be equally valuable. I would start by splitting up the answer into several components:

1. Version control & collaboration
2. Data format
3. Visualization
4. Integration

Version Control & Collaboration

The question here is whether git is a good solution to collaboration, and I would say yes, certainly. Assuming you can solve the other components to this problem, git provides several advantages that I would see applicable to genealogical research:

• The ability to share data between researchers.
• The ability to 'branch' a body of work, to experiment with new data to see how it applies to a record(s) before committing the change.
• (Specifically for GitHub/Bitbucket) The ability to submit pull requests between efforts. This gives two (or more) genealogists the ability to reason about changes to a series of data before settling on the 'master' result. This is probably the single greatest win git provides above other version control systems.

To that last point, I would think there is an advantage to version controlling a genealogical research effort with any kind of tool you can get your hands on. At this time I would think git one of the best options available.

Data Format

Data formats are an implementation detail. It is what you do with that data that is more important than how the data is stored. In order for systems like git to manage the data well, a text-based format would be necessary. Formats like GEDCOM, XML and JSON have their own (dis)advantages, so without getting into a religious argument I would simply offer this advice: pick one and move forward. If it is done right, any solution should be able to migrate from one underlying format to another without the user being the wiser.

Visualization

This horse has been beaten dead, and is not worth revisiting at this point. There are many ways to parse through genealogical data, browse it, and modify it. A solid solution here is essential to make sure non-technophiles are able to use whatever system is being offered.

Integration

This is the million-dollar issue. Can you make a front-end that works seamlessly with some text-based file format, and have it manage everything with a version control system on the back side? If you think of a researcher's data as code, what you are talking about here is a genealogical IDE, and something that would be a huge tool to have in one's toolbox. Open source IDEs exist (Eclipse comes to mind) that I would think can be extended to manipulate and visualize genealogical data. The git integration is already there, so it is a question of tying it all together.

Answer 3

Genealogists will never use git directly, but a frontend is eminently possible.

Neither XML or (to a lesser extent) json map well to a line-based version control system (because it can be hard to retain the structure when a individual lines are changed by different people at the same time).

The only real problem with using git for version control of hierarchical text-based data formats is merge conflicts due to changes in adjacent lines. These can be easily resolved by parsing the full surrounding data entity of those lines and determining whether they actually differ in the two branches.

In any other cases when two commits change the same line, you want there to be a merge conflict so the user has to change their data (or rebase/merge upstream, but that would be a ton of work) in order to make it merge cleanly.

To enable that property, you need the text format to keep records sorted and indented identically so there are no superfluous changes with almost guaranteed merge conflicts.

For this to work with a format where records are referenced by ID (probably all of them), you need that format to use UUIDs for internal IDs. (Otherwise there will be conflicts due to sequential ID generation.)

Gramps XML meets all the requirements for this.

Gramps could be extended to mirror each database as a git repository containing the uncompressed XML export of the database. You will only need to export or import when doing checkout, branch, pull, push, or merge operations; for the rest of the time Gramps will still operate on its internal DB.

Genealogists are never going to host their own git servers, so you would need to make a centralized web app where database administrators can review and merge changes. For any commit, just load the entirety of each of the two versions of the file as an object in memory, diff the two versions, and display the difference in a human-readable format.

Genealogists can then update their local copies by clicking a "pull" button. You'd want to keep a separate branch solely for tracking the remote so you never have merge conflicts with that, and then when the genealogist pulls new changes, branches with merge conflicts against the new changes could be displayed.

(Use pygit2 or the like.)

Answer 4

I've been thinking along the lines of using git for genealogy data for the genealogy program that I'm considering writing. Obviously, using GIT would require a front-end that is user friendly and specifically designed for genealogy usage.

The attraction is to be able to say "this fact about ancestor D was entered by user X." But there is a bit of a split in philosophy when what we want to record is "based on these three questionable sources, user X has concluded that ancestor D was born in XYZ"

In the git world, and in all other source code control systems that I've used over the decades, the guiding principle is that there is one correct way for the lines of code to be. Code has a bug, and this patch fixes it. Or the code was missing this feature, and this patch adds the feature.

In genealogy, we often want to record stuff like "while based on these three questionable sources, user X has concluded that ancestor D was born in XYZ, user Y believes that these two additional sources make a strong case that ancestor D was really called E, and he was born in UVW"

I am not sure that using git (or any other source-code oriented tool) will really help us do what we want.

Answer 5

Atomicity

In versioning there is the concept of atomicity.

So you do not push in a Gedcom file because then your are managing gedcom files not persons equally as you do not push a word document in a versioning system and call this "requirements". No, you push in every atomic requirement as a "thing".

I think every research-on-a-person is 1 repository in git. e.g. on github (with its own issues board, wiki, forks and .. releases of a steady state of a research). So every person is 1 repository / 1 project.

Manage changes

These tools are about helping you to manage changes.

So your branch does not represent the changes in that person's life. So you will not have subbranches as his/hers children. Because the thing you are trying to manage is your research on that person. So the changes are "new information that comes in". So if you find out that the death date is different that is a new change in your project.

Content

Your main branch consists of folders labelled per date-time in that life. Each folder contains a file describing what happened on that date-time in that life. Each folder has an optional label that can be standardized for standard events: birth / dead / baptized / etc.

Things that are uncertain are development branches on the main branch e.g. "researching death date"

You can then link to other repositories for related family. So if a child is born you link to another repository inside that file in that specific datetime folder. I think that file can just be a HTML file. Since there can be header meta tags added for specific stuff and it makes it handy to render photos and such (also e.g. the REL tag on a URL to describe the relation). Localization can be done with pot files but probably just making a index_Nl_nl.html is easier.

The link can point to loadbalancers alike sites that display all repos of that person or to just 1 specific repo (e.g. another one that you own).

Analysis and Reporting

Crawlers can trace through reposities to create search engines and whatever ui tooling.It is the task of the Crawler to "type" the information (give it a type) and such so not the other way around. Like Aristotle said: we have the history and the categorization/modelling based on the history and this scope is the history.

Forks

Every researcher has its own repositories. So if 100 persons research "Jane Doe" then 100 people will have such a repo. In some cases this will be based on a fork of someone's else's repository which makes it easy. In other cases it will be different fork threads running next to each other until one survives. Exactly as is currently on Github with many projects making exactly the same software but some more popular than others.

To tie forks together is technical git stuff but would be Geni.com alike where everyone can link in someone else and create in the end gigantic networks but different than in Geni where you have to "open" one profile, here multiple people can make branches / forks. So someone does not have to "open" a profile you can just fork it and make your changes etc.

But who can provide a platform with hundreds of thousands of Git Repo's for free?

You can create repositories for free on Github but i do not know how they will react if you would create 200.000 repositories at once (since i think the average Gedcom file nowadays contain quite some people). Maybe someone can check with Microsoft if they want to, next to linkedin also create a bloodedIn site with a stripped down version of Github offering each user tens of thousands of repositories and a stripped down version of bing to search only in these repo's.

Could be tested: https://github.blog/2019-01-07-new-year-new-github/

Smaller tests can be setup in an hour

I think this can start right away since its pretty simple.

• Create a repository for each person (or auto generate: https://docs.github.com/en/rest/reference/repos#create-a-repository-using-a-template)
• Then inside (content) add folders with an iso date + label.
• Then in these folders a html file with iso language extension.
• Then in the html file whatever html with links with rel tags to the other repositories.
• A html file in the root with links to all datetime folders for a person would be handy to autogenerate, it would be nice if it also could accumulate the links in the html files (rel=daughter).
• I think https://pages.github.com/ auto generates a site for you based on this