It means that when he was growing up nobody liked him.