The GEDmatch Relationship Tree is a wonderful idea the imaginative people at GEDmatch put together. No, there is not much documentation about it and it is still said to be a "work in progress", but it has been there for a number of years and does something few if any other DNA analysis tools do.
It uses the two people that you give it, whose relationship you want to determine. It uses both the autosomal sharing levels and X-DNA sharing levels to determine relationships.
It starts off by building up the tree for the first person. It adds the father and mother. Then it adds their grandparents. Then their great-grandparents and it keeps going as far as it needs to. When it adds each pair of parents, it add a father and a mother, so it knows their sex as it adds them.
Then for each ancestor, it determines the average autosomal shared and the average X shared with the original person. For autosomal, it lists parents as sharing 3587 cm, grandparents half of that (1793.5), great-grandparents half of that (896.75), etc.
For the X chromosome, it uses X-DNA rules. It starts with the first person having 196 cM. If the first person is male, then his father shares 0 and his mother shares all 196 cM. If the first person is female then each of her parents share on average half or 98 cM each. It continues this pattern up the tree depending on whether the person is male or female.
Once it calculates the autosomal and X the first person shares with an ancestor, it then takes the ancestor, and works its way back down to get to the second person. It asks you for the generational level of the 2nd person relative to the 1st person so that it knows how many generations to go down.
And it does the same thing in reverse. For every ancestor, it adds a male child and a female child. For every child, it calculates the autosomal as half shared of the parent.
For each female child, the X is calculated as half the parent. For each male child, the X is calculated as 0 if the parent is male, and the full value of the parent if the parent is female.
Once it reaches the 2nd person's generational level, it will then compare the autosomal shared and X shared with the amounts shared that you entered as input. If the values are within the "overlap factor", then that relative is considered a possible candidate. The default overlap factor will include anyone where your entered values are between 29% below the projected value or 41% above the projected value.
It will then draw the relationships it finds in a very nicely done chart that contains all the average shared matches for autosomal and X that it calculates.
If no relationships are found, you can increase the overlap factor. An overlap factor of 2 goes from 50% below to 100% above.
Below is an example of a run I did, where I compared myself (male) with an female DNA tester that I don't know how I'm related to, but we share 41.9 cM autosomal and 27.9 cM on our X. Level 1 didn't give me any results, so this is run with level 2 which gives two most-likely relationships.
Because of the variation in DNA that descendants get from their ancestors (it's not exactly 50% except for parent-child), the relationships given are in no way guaranteed, but they are the ones that are quite possible and are most likely as far as what the averages say.
In my case below, the tool tells me that the most likely ways I relate to this person as a 3rd cousin are through the following two connections:
My mother's father's mother's sister's son's daughter's daughter.
My mother's father's mothers' sister's daughter's son's daughter.
Both give expected shared autosomal of 56 cM and X of 18 cM compared to the 42 cM and 28 cM actuals.