Maybe it was also an ideal for men to separate the aristocracy from the working class? Rich gentlemen could sit inside and read and be pale and wan while poor men had to work in the fields, or other menial jobs where you are outside (which would make them bigger, more muscular, and tan).