the reason they were mean to him I believe, is that Kadgar disagreed with the Kirin Tor (the school of magic in Dalaran) and abandoned his studies after showing a affinity and talent for magic