To be fair, I don't think most guys would actually hold it against you. And not for any loftier reason than most people empathize with their own gender first. Whether or not that's good for society as a whole, it seems to be most people's default mode.