You are close in the sense that dwarf refers to a specific condition, dwarfism, and not all little people have dwarfism. However, midget is just an outdated term that has connotations that are not positive so a lot of people want to distance themselves from the term.