No, you just have to laugh because it's a funny misunderstanding(your phrasing makes perfect sense as I get asked this all the time and I say the city I grew up in) and say, I mean are from (insert city you are currently in)? It's only awkward because you assumed guilt for no reason. You asked the right question.