While there is a difference in “punching up and punching down.”, that difference is not that one is racist and the other isn’t. A joke based about race is still racist. You can debate that one form of racism is more acceptable if you want, but you’re still debating racism vs racism.