I think Jon couldnt say it better in the scene where he discussed the matter with the brothers. "we can learn to live with them, or we can add them to the army of the dead". No one can stop zombie Wun Wun breaking those gates