I think she's saying that before he left his wife for his step-daughter, his work wasn't being compared to him as a person. So in 1979 when Manhattan came out, people could have thought he was a perv because of his work, but that's not what they were saying UNTIL he got with Soon-Yi.