I think you mean “serious threat to anything South of the Wall”, no?

Do you mean South of the Wall? It seems the White Walkers are plenty threatening on their own side of the Wall.

"the White Walkers/Others probably aren't a serious threat to anything North of the Wall."