It just strikes me as fishy. If Varys told Elia "Here, we're going to swap this baby in case someone comes to kill you." Why would her reaction not be "Why don't you get me, my daughter, and my son out of here now?" The woman was a Martell—she wasn't an idiot.