Well by most people's definitions you are having an "emotional affair" but that's hardly the same as an actual affair (i.e. something you need to confess to and should never be allowed) however it does indicate that both your marriages are in serious trouble and you both either need to fix them or get out.