Because the purpose of the attack was to get south of the wall and safely behind it, away from the Others/White Walkers.