Women are called females while men are called men all the time. Women are called girls while men are called men all the time. It's just about universally accepted that calling a man 'boy' is derogatory even though it is perfectly socially acceptable.