If it’s taking 5 minutes to load Starfield, you need to check where it’s installed. That sounds like you’ve got it on an external USB drive. Either that, or something else is throttling your performance.
On my Series X, it takes less than a minute to start the game from the main Xbox screen (not Quick Resume), select…