No, he is probably not afraid of one of them raping him. I will bet he is afraid of one of them making a failed pass that at best screws up his relationship with his wife, or at worst ends with false legal accusations against him from one of his female coworkers though.