That’s what that “Decline” button is for. I’m totally ruthless about managing my schedule and regularly refuse to go to meetings that conflict with my lunch or other meetings I’m in. I had a colleague that didn’t quite understand Outlook and when I’d decline, they weren’t able to figure it out.