You hit the nail on the head in your opening sentence. I don’t remember where I first heard it, but I love the line “my freedom to swing my arms ends at your face”, because it succinctly explains why we lose our freedoms when they cause harm to others.