Haven't hurt anyone? They were controlling and manipulating humanity the whole time they've been on Earth. They're like the alien version of big bankers. Plus, it's the idea that you're being controlled or enslaved by an unseen force. They weren't predators, so the threat seems innocuous, but it is a threat.