The family tree of ancestors is a binary tree (although as you move up, you will start to see nodes that are identical -- this is pedigree collapse). So if you think of each ancestor as making an equivalent "contribution", then yes, the total contribution from ancestors from one places should be a dyadic rational number.
But Ancestry is determining ethnicity by analyzing DNA, not by analyzing the family tree. Each parent contributes half of the DNA to a child, but already at the level of grandparents, it is usually the case that some of the grandparents contribute more DNA to the grandchild than others. So even if it makes sense to model the percentage of ethnicity as a sum of contributions from each ancestor at a certain number of generations back, the resulting distribution for the ethnicity is not restricted to dyadics.
Regarding the mathematics of genetics, I'd love such a source myself. But the first thing to do before getting into the math is to learn all of the complications that would go into building reasonable models of DNA descent. For instance, when you look at the Ancestry DNA data, you will find that segments have lengths given in units called centiMorgans (cM). DNA is measured at sporadic places along its length. If two people have exactly the same sequence of A, T, G, or C along some portion, then the length of that segment is measured and reported as a match. You might guess that if two segments cover the same number of positions of DNA, then they will have the same cM. But that is not the case. First of all, because the positions of measurement are not equally spread, a match at 500 consecutive positions on one segment may actually be over a longer strand a DNA than a match at 500 consecutive positions on a different segment. Furthermore, it is known that some areas of certain chromosomes recombine faster than others. If two people match on a segment that recombines very quickly, you may expect it to come from a more recent common ancestor than a common segment with the same number of measured positions and same amount of DNA which is known to recombine more slowly.
Then there's the problem with how DNA is measured. At each measured position is recorded a nucleotide reading on each of two corresponding places on paired chromosomes. The result might be AA, or AT, or even ?C (where ? means "no-call", meaning the measurement failed to give a result). If you look at the raw data file, you may see three consecutive positions recorded as AA, AG, and CT. That does not mean that one chromosome showed AAC while the other showed AGT. The measurement process does not know, for a given pair, which chromosome produced which member of the pair. So it can often happen by chance that over a huge number of potential matches and a huge amount of DNA in each match, you can find a sequence of 500 consecutive positions or so where one person has a letter that matches one of the two letters of another person at the same positions, but where the letters that match at each position are actually jumping back and forth between the chromosomes. These people would be declared a match, even though the match isn't arising from a single strand of DNA. This is is a false positive, called Indentical by State, as opposed to Identical by Descent.
Another complication is introduced by endogamy, which causes significant pedigree collapse. If a person's parents are siblings, for instance, then that person got all of his DNA from just two grandparents instead of four. Thus, even though they are two generations away from those grandparents, the expected DNA contribution of each grandparent to the grandchild is instead comparable to what a parent would provide. Using a regression algorithm or the like to predict how closely related the grandchild is to a grandparent would overestimate the closeness of the relationship, predicting a parent/child relationship instead. This is an extreme case, but endogamy causes lots of problems with predicting relationships in common practice. For instance, it makes it significantly harder for people with Ashkenazi Jewish heritage to use DNA to test hypotheses about their family trees.
I haven't investigated the ethnicity algorithms -- I spend my time trying to use DNA to identify cousins and find common ancestors. But one reason I haven't approached ethnicity is that I've seen lots of anecdotal accounts that ethnicity results aren't reproducible -- testing the same person at different companies gives very different results. So you should take the percentages you gave with a grain of salt.
The main concept in finding cousins with DNA is predicting relationship closeness from DNA results. The two approaches I see people use are gathering data from pairs of people about their relationships and their DNA comparisons to build distributions DNA matches for each type of relationship (Blaine Bettinger has done the most visible such analysis). The other is selecting a model for how recombination happens, creating an initial population, and simulating the propagation of DNA through generations. In the simulation, you will know how each person is related to each other.
Both have their drawbacks, but I'm prone to trust more the simulation method. The data used in the first method is self-reported, from different companies with different standards, and people are not necessarily using good standards of proof for the relationships they are submitting. Garbage in, garbage out. But perhaps for some of the closest relationships, the data may be pretty good. The second method is only as good as the model of recombination and coding in the simulation, and that's the sort of thing I'd like to find a good book about.
