It could be referring to Rick and his people, who are basically as dangerous and far gone in the eyes of the townsfolk as the walkers are.