My immediate thought was how people at that level of society had (and still have?) a certain way to do everything; It certainly would not be proper to take off one's hat right then and there. It has to be done in the privacy of your bedroom, and probably by your lady's maid.