Maybe my mind is playing tricks on me, but I thought Varys *did* have to escape with Tyrion, but came back for his appearance in the Epilogue of Book Five.