There are much fewer roles open to people of color to begin with so when you take away one like this it does more damage. When a white actor has trouble finding roles because almost every part is defaulted to another race you can then make that argument.