SPOILERS….Its been a while since I saw it but I'm fairly certain he got burned and in a fit of rage killed his wife and baby. He wanted to believe they just left (and you may even see it at one point), but he knew they didn't, and that's why he could never go inside the house. He knew what he'd find there.