I can't believe i'm defending this shows logic… but he does say in his first episode why he remembers the other world. The curse is to take away everyone happiness so his curse is to remember the past world and have his daughter live right next door with no memory of him to torture him.