It wasn't right when they were together in season 3, when Jess is obviously flawed to the point of being unhealthy… But when he shows up a couple times in later seasons, with his act together and looking fine as ever… That's when you realize they were right.