I’m not sure he’s being racist so much as he’s saying that people should stop making such a big deal out of things. He’s from the generation that didn’t talk about feelings, and didn’t discuss social issues like this. They “kept it quiet” for the sake of propriety.