In American culture, I would agree with you that the distinction of who is marginalized and not marginalized is pretty clear cut. What I meant to communicate is that the intent to marginalize is there with any racist act, but that the ability to marginalize may not be depending on historical precedent.