Exactly. He has the right to express his opinions and ideas without the government punching him in the face. The rest of us are free to do it.
Freedom of speech means freedom from government prosecution because of speech; it doesn’t mean freedom from societal consequences.