I don't think it's as much assuming the worst of Connor as much as Oliver knew who Connor was when they met (the arrogant playboy) and expected him to react a little more angrily to a breakup because breakups hurt and some people crumple but others—like he expected Connor to be—lash out.