My theory on ESB is that the hyperspace trip was quick, but hiding in the asteroid field lasted a long time. After all, the Empire got to Bespin ‘right’ before the Falcon did, which implies that fully functional Star Destroyers weren’t going much faster than Han and co.
That would be a good explanation of why Vader was…