If you are holding up traffic behind you then you are being the jerk. No matter how fast you go. This is the “everyone who wants to speed faster than me is an asshole” mentality. In some places you are probably also breaking the law by impeding traffic in the passing lane.