Doesn't seem so hard to me. "The End of Time" took place while the Doctor was in the desert talking to the Moment. When he got back to Gallifrey, no one thought to mention, "Hey, Rassilon tried to teleport the planet away before you could blow it up to buy himself enough time to destroy the universe, but your future…