Based on the information you have provided, I don’t think you are quite on the right track. You state that:
I believe that one of my female second cousin's uncle is my paternal grandfather.
However, you could not inherit any X-DNA from your paternal grandfather. All of your X-DNA comes from your paternal grandmother and maternal grandparents. You inherited an X-chromosome from each of your parents (red and green on the diagram below). The red X came from your father, which came from his mother. The green X came from your mother.

You can see from this diagram that you cannot have inherited an X from your paternal grandfather, because he only passed on a Y to your father.
Based on the information given, you could be related to your supposed second cousin on either your paternal grandmother or maternal grandparents' lines:

The fact that your brother shares no X-DNA with the second cousin makes it perhaps more likely that you are related through your paternal grandmother, since your brother inherited no X-DNA from her, while you did. It is equally possible that he just inherited a different X-chromosome segment from your mother, and that you are related to your second cousin via your mother. It is very difficult to say anything more specific without the particulars.
My final comment is on the fact that you keep referring to your DNA match as a "second cousin". The relationships you describe in your question ("second female cousin's aunts or uncles is my grandparent") would actually make her your first cousin once removed. You do not state the length of the match, but keep in mind that there is very wide range of match length for a second cousin (43-504 cM (average 238 cM) according to the Shared cM Project). Therefore, keep in mind that you might not actually be second cousins, you could (for instance) be related less closely and happen to share more DNA than average for your degree of relationship.