Usually, everyone else is standing up too... so yes, they will be next to your seat, but you stand up in front of them and make them wait, I do it all the time. I remain seated and I’ve NEVER had someone behind me on the plane get off in front of me.