Explain Jon's reaction to Little Finger please. I didn't grasp his anger.

"I wasn't bothering anyone."