It depends on what style of swastika you use. If you're using a black swastika, bent clockwise at a 45 degree angle, in a white circle, on a red background, yes, you should absolutely stop using it. Or keep using it, whatever, but don't try to bullshit the rest of us into believing it's a religious symbol.