I'm not sure it's a "fuck everyone who isn't us" attitude. Almost everyone in the group now was "not us" at some point. I'm sure as he sees it, if he saw the need to take Alexandria over, he'd be doing everybody inside a favor.
(And he's probably right.)