I just wrote my own family tree using TikZ. For each male and female entry i used a rectangle, colored pink for female, blue for male. However, this does not show up if I am printing in black and white. I was wondering if there was any standard or recommended notation for how to represent a person's sex in a family tree. I thought of using the Mars and Venus symbols as backgrounds in the box.
A square for males, circle for females, is the standard for genogram family diagrams, used in medicine, genetics and social work.
However, these diagrams are rarely used in genealogy (so far). Genopro is one of the few programs that does generate them.
Although genograms can use colours on lines to indicate types of relationships, they are also slightly different line patterns. If you used more features of genograms, beyond the simple gender indicator shape, they would work best with colour but you would still be able to use black and white copies.
Here's a short lesson from FamilySearch on Tracing Family Traits Using a Genogram (video and pdf) that has the basics how to create genograms, and more references. With genetics increasingly becoming a part of genealogy, genograms may become more used in the future.
Genograms normally have the names outside of and below the square/circle rather than inside, see some examples. That may be too big a change to the look of your chart though.
The rounded corners convention is the other option if you need boxes to put content in (see an example). I don't think it's ever been described as a formal standard, but it's not unknown.
The Mars/Venus symbols were used in medicine in mostly Europe in the early 20th century but I haven't seen them used in modern genealogy charts, they're not a standard or convention. It may be difficult to see the difference between them at small sizes (which is also possibly a problem with the rounded corner rectangles). See a discussion of historical pedigree typography (pdf).
In summary, the squares and circles of geneotypes are an international standard but it's not widely used in genealogy (possibly because it can look quite ugly with the text outside). The curved box corners for a female does exist as a convention, and is easy to do. Using Mars and Venus symbols in the background isn't a standard or convention and might make the text on them hard to read.
No symbols are used in a "standard" family tree, since 99% of names are either masculine or feminine specific. Symbols are typically used when the person's name is less important, such as in heraldry and genograms.
In heraldry, males have shields and females have pendants, or sometimes males have pointed-bottom shields and females have rounded-bottom shields.
In genograms, males are squares and females are circles. But they are often presented without names, so they can be reviewed and studied anonymously.
I've seen the Mars and Venus symbols being used more often on foreign sites, but historically they were probably viewed as pagan symbols.
Some modern charts use blue for males and pink for females, or rectangles for males and ovals for females, but there is little value aside from aesthetics. Depending on the size of the chart, these artistic changes can either add or subtract complexity for the viewer.