Also, even after all of Robb's armies were left in the South, both the Umbers and the Boltons were able to amass fairly large armies in the North, each numbering in the thousands, with another round of levies, so obviously not al of the fighting men went with Robb.