Except it's not even close to being the end of the story. Gene expression varies. There are all sorts of scenarios in which your phenotype might not match your genotype. For example, if your cells don't recognize androgens than you may very well have a Y chromosome but be born with a vagina.