If the parasites are capable of creating unhappy memories (which makes sense, it's hard to imagine that a species of telepathic parasites would get very far in the universe if they had a tell so obvious that Morty could figure it out), then any of the family could have been the parasite. It hardly seems logical that…

Hang on.