If he had said, “¡a la madre!” or even a sublinguistic “ay” or “esa”, you would be correct in calling it brown. There are any number of idioms and exclamations and inflections and dialects that would immediately and accurately point to a black speaker.