He does have reason in wanting to make the move fast the longer he waits gives Ramsay more time to prepare for a full blown attack such as he would give Ramsay the time to create anti siege equipment or gather supplies for a siege it even gives Ramsay the chance to gather more men too his side.