Speaking as an Asian-American, the reason is because racism is about power. Because power in American society disproportionately favors whites, when a white actor takes a role erasing a character written as Japanese, it is fundamentally different than when the same swap is a black actor.