I don't think Rigby was worrying about we "don't hang anymore" I think he was worrying he was a going to burden Mordecai and relationship with CJ. rigby knows he a fuck up he doesn't want to be one but, he is. As much as he wants to bro out with mordecai he doesn't want that bro time to cost Mordecai. He doesn't want…