I’m pretty sure the Blackfish was referring to Jaime’s oath to protect King Aerys. Jaime has been called Kingslayer and Oathbreaker since before this show started. That kind of thing gets you on the shit list with just about everybody, regardless of allegiance.