In the books, Castle Black is huge but mostly in bad condition. The Night's Watch used to be orders of magnitude larger, and in their depleted condition have been unable to man most of their castles, or even to fill the ones they do man, let alone maintain them.