That's true. We have no evidence that suggests these unions were rape, nor do we have evidence that suggests otherwise. All we have is evidence that they bore children together, and that the children survived.

So, why does it matter that Homo sapiens had sex with Neanderthals? It matters because now we have genetic evidence that humans as a species don't always kill and destroy what we don't understand. Sometimes, we fall in love and settle down instead.