But Oliver found out about the affair back during "State vs. Queen" when Moira told him and Thea about it, and to my recollection he didn't react very negatively then. That's how Felicity knew about it in the first place, it came out during the trial!